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This  Hnal  Technical  Report  covers  research  carried  out  by  the  Parallel  Systems  Laboratory:  Ac¬ 
cess,  Allocation  and  Control  at  UCLA  under  DARPA  Contract  Number  MDA  903-87-C-0663, 
covering  the  period  July  1, 1987  through  June  30, 1992. 

Below  we  restate  the  tasks  which  have  been  the  subject  of  this  research  effort.  We  then  give  a 
summary  of  the  salient  results  of  this  research  in  capsule  form  which  is  followed  by  an  extensive 
list  of  publications,  dissertations  and  theses  which  have  been  the  result  of  this  research.  The  ctm- 
tract  period  covered  is  from  July  1, 1987  to  June  30, 1992;  however,  one  must  note  that  whereas 
this  contract  formally  ended  on  June  30, 1992,  the  period  from  July  1, 1990  to  June  30, 1992  was 
simply  a  no-cost  extension  of  frmds  to  complete  and  publish  results  of  studies  that  had  begun 
during  the  main  contract  period. 

The  included  papers  summarize  a  number  of  our  research  activities  but  are  only  a  sample  of  our 
work.  Of  course  the  more  complete  results  are  contained  in  the  full  set  of  publications  we  have 
listed,  copies  of  which  have  been  sent  to  the  DARPA  contracting  office  during  the  period  of  this 
contract 

Brief  Statement  of  Work 

The  brief  statement  of  wwk  which  describes  the  tasks  on  which  we  have  been  woridng  during 
this  contract  are  as  follows: 

TASK  /.  FUNDAMENTAL  TOOLS  AND  ISSUES  IN  PARALLEL  PROCESSING  SYSTEMS 

It  is  imperative  that  we  develop  evaluation  tools  as  well  as  fundamental  understand¬ 
ing  oi  the  behavior,  performance  and  tradeoffs  for  parallel  processing  systems.  We 
propose  to  develop  such  tools,  and  to  evaluate  the  way  in  which  the  system  resources 
interact  widi  each  otha*  and  with  the  demands  placed  upem  them. 
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TASK  IL  PARALLEL  PROCESSING  ARCHITECTURE  AND  ALGORITHM  DESIGN 


A  given  set  of  resources  can  be  configured  to  be  centralized  or  distributed  in  many 
ways.  We  propose  to  study  the  effect  on  performance  and  cost  of  various  system  to¬ 
pologies  (e.g.,  series,  parallel,  series/parallel,  network),  organization  and  architecture. 
We  will  design  algorithms  to  operate  in  a  parallel  processing  environment  and  will 
evaluate  the  effect  of  unreliable  processors. 

TASK  III.  MIGRATING  AND  ADAPTIVE  PROCESSES 

The  distributed  control  of  parallel  systems  has  proven  to  be  a  major  development  in 
system  design.  We  propose  to  study  the  design  of  distributed  algorithms  which  adapt 
to  the  load  and  structure  of  their  environment,  and  to  evaluate  their  performance  (de¬ 
lay,  efficiency,  complexity)  in  massively  connected  systems. 

TASK  IV.  DISTRIBUTED  DATA  BASE 

The  coordination  of  a  data  base  in  a  distributed  environment  is  an  extremely  impor¬ 
tant  issue  in  large  systems  today.  We  propose  novel  methods  for  accessing  and  up¬ 
dating  data  base  systems  from  distributed  sources  using  the  technique  of  optimistic 
locking;  this  algorithm  has  the  potential  for  providing  higher  efficiency  in  a  variety  of 
complex  systems. 

TASK  V.  TIGHTLY  COUPLED  PARALLEL  PROCESSING  SYSTEMS 

Tightly  coupled  processors  in  a  parallel  processing  environment  may  suffer  serious 
performance  degradation  problems.  We  propose  to  provide  tools  for  evaluating  per¬ 
formance  for  a  very  wide  class  of  parallel  processing  systems.  We  will  include  the 
effect  of  the  number  of  processors  available,  the  parallelism  inherent  in  various  gr^h 
models  of  computation,  the  arrival  process  of  new  jobs,  and  the  randomness  in  the 
task  computation  times.  A  related  issue  is  that  of  massive  connectivity  in  massively 
parallel  systems.  We  propose  to  evaluate  the  tradeoff  among  communications,  pro¬ 
cessing  and  storage  in  these  systems. 

TASK  VI.  EXPERIMEHTATION  IN  PARALLEL  SYSTEMS 

A  key  t««lc  in  this  project  is  to  conduct  a  carefully  staged  sequence  of  experiments  in 
parallel  processing  using  the  existing  testbeds  available  in  the  Computer  Science 
Department  at  UCLA.  We  have  named  this  the  Benevolent  Bandit  Labtxatray  (ot 
BBL  Project,  fm-  short).  Our  main  purpose  is  to  use  it  as  a  vehicle  fcv  experimenting 
with  an  operational  parallel  processing  system;  initially,  we  intend  to  use  it  to  harness 
the  power  of  underutilized  workstations  which,  fortunately,  ate  already  connected  to¬ 
gether  as  a  "system”  through  a  local  area  netwtxk. 
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Brief  Summary  of  Significant  Results 


During  this  contract  period,  we  have  published  21  papers  and  have  had  2  additional  papers  ac¬ 
cepted  in  profession^  journals.  Moreover,  we  have  contributed  2  book  chapters.  In  ad^tion  to 
all  of  this,  6  Ph.D.  students  have  graduated  as  a  result  of  the  work  they  have  conducted  under 
this  research  in  addition  to  6  Master’s  students. 

Our  major  results  fall  into  three  somewhat  overlapping  areas  which  relate  to  the  statement  of 
work.  These  three  areas  are: 


1.  Parallel  processing  systems; 

2.  Distributed  processing  systems; 

3.  Distributed  communications 

In  the  area  of  parallel  processing  systems,  we  have  a  number  of  interesting  results  and  develop¬ 
ments.  One  of  our  major  activities  during  this  contract  period  was  to  develop  what  we  call  the 
Benevolent  Bandit  Laboratory;  this  is  a  testbed  for  understanding  the  behavior  of  distributed  al¬ 
gorithms  in  an  environment  in  which  we  recapture  the  idle  processing  capacities  from  a  large 
number  of  workstations  connected  in  a  local  area  network.  The  concept  here  of  course  is  to  try 
to  take  advantage  of  the  enormous  amount  of  idle  time  experienced  by  most  woricstations. 
Indeed  we  built  a  shell  around  the  woikstation  operating  system  which  senses  when  dw  proces¬ 
sor  is  idle  for  more  than  a  minimum  amount  of  time  at  which  time  our  Benevolent  Bandit  shell 
takes  over  the  processor  and  runs  a  background  process;  as  soon  as  the  user  of  the  system  needs 
access  to  his  own  workstation,  the  shell  immediately  relinquishes  control  and  returns  it  to  the 
user.  We  implemented  the  system,  we  measured  it,  we  wrote  and  ran  applications  on  it,  and  we 
analyzed  the  behavior  of  this  system  and  of  a  large  number  of  other  similar  systems  of  much 
greater  extent  both  in  numbers  and  operations.  Indeed  the  paper  by  Kleinrock  and  Korfhage 
"Collecting  Unused  Processing  Capacity:  An  Analysis  of  Transient  Distributed  Systems"  which 
is  included  in  the  Appendix  of  this  report  summarizes  the  analytical  results  which  we  obtained 
for  this  system.  The  bottom  line  for  such  an  operation  is  that  one  can  capture  the  unused  capacity 
of  networked  woricstations  and  that  the  performance  is  predictable  and  understandable. 

Another  majw  development  we  achieved  in  our  study  of  parallel  processing  systems  was  to 
understand  how  many  processors  one  should  use  in  raxler  to  effectively  carry  out  the  dependent 
tasks  of  a  sequmce  of  jobs  as  they  flow  into  a  parallel  processing  system  with  P  processes.  The 
problem  here  is  if  we  have  too  many  processes,  the  efficiency  of  each  is  low  (but  the  response 
time  is  conveniently  short);  on  the  other  hand,  if  we  have  too  few  processors,  the  efficiency  of 
each  is  high  (but  the  response  time  degrades).  Using  a  combined  objective  function  known  as 
power,  which  is  simply  the  ratio  of  efficiency  to  mean  response  time,  we  determine  the  (primal 
number  of  processors  (Mie  should  use  in  such  a  system,  determine  the  optimum  load  such  a  sys¬ 
tem  should  sustain,  and  identify  the  principles  of  operation  and  insights  which  come  out  of  such 
an  analysis.  The  key  ideas  here  are  summarized  in  the  ptq)er  by  Kleinrock  and  Huang  entitled 
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"On  Parallel  Processing  Systems:  Ahmdal’s  Law  Generalized  and  Some  General  Results  on  Op¬ 
timal  Design"  included  here  in  the  Appendix  of  this  report. 

A  third  major  thrust  which  was  begun  toward  the  end  of  this  contract  period  was  to  study  the 
behavior  of  asynchronous  distributed  processing  in  a  moderately  tightly  coupled  parallel  pro¬ 
cessing  system.  In  particular,  the  notion  of  decomposing  a  job  into  a  number  of  parallel  tasks, 
each  one  of  which  runs  on  a  separate  processor,  but  each  one  of  which  is  not  totally  independent 
of  the  others  was  analyzed  in  some  preliminary  studies.  There  are  two  ways  to  run  such  a  sys¬ 
tem:  asynchronously  (the  method  we  favor)  and  synchronously  (a  more  conservative  approach). 
We  have  compared  the  maximum  improvement  available  for  asynchronous  operation  versus 
synchronous  operation  in  the  paper  by  Felderman  and  Kleinrock  "An  Upper  Bound  on  the  Im¬ 
provement  of  Asynchronous  vs.  Synchronous  Distributed  Processing"  and  found  that  the  gains 
were  clearly  predictable  and  of  some  significance.  Furthermore,  we  analyzed  the  details  of  a 
specific  two-processor  system  in  the  paper  by  Kleinrock  "On  Distributed  Systems  Performance" 
and  gave  an  exact  analysis  of  the  behavior  of  this  system,  demonstrating  the  speedup  possible  in 
such  an  environment.  Both  these  papers  are  included  in  the  Appendix. 

In  addition  a  number  of  studies  involving  other  aspects  of  parallel  processing  systems  was  stu¬ 
died  and  reported  upon.  For  example  we  determined  the  algorithms  for  performing  optimal 
parallel  merging  and  sorting  using  a  limited  number  of  processors  in  an  efficient  fashion.  We 
also  took  advantage  of  broadcast  communications  in  such  sorting  algorithm  problems.  (Certain 
aspects  of  load  sharing  in  limited  access  distributed  systems  were  studied  for  a  collection  of 
loosely  coupled  parallel  processors.  Moreover,  we  studied  the  behavior  of  load  sharing  among  a 
set  of  processors  in  a  broadcast  network;  we  found  that  if  tme  balances  the  most  heavily  loaded 
processor  with  the  most  lightly  loaded  processor,  significant  gains  can  be  had  and  it  takes  very 
few  such  pairs  to  be  matched  before  the  majority  of  the  improvement  is  achieved;  this  occurs 
long  before  all  processors  ate  matched  to  share  load  thereby  resulting  in  a  very  efficient  load 
sharing  at  very  low  overhead.  A  related  load  sharing  and  load  balancing  study  looked  at  a  pro¬ 
cessing  system  as  a  "field"  of  processors  which  dynamically  share  load  based  on  the  immetfiate 
backlog;  here  again  the  effect  of  load  sharing  was  shown  both  analytically  and  simulation-wise 
to  be  extremely  effective. 

Our  second  major  area  of  research  was  in  the  field  of  distributed  systems.  This  area  covers  a 
variety  of  systems  models  including  the  problems  of  multiaccess  in  broadcast  communications 
systems,  distributed  database  systems,  and  distributed  search  methods.  Basically  we  have 
developed  a  number  of  advanced  multiaccess  schemes  whose  perfcmnance  is  found  to  be  quite 
efficient  and  whose  analyses  have  been  carried  out  to  give  excellent  predictions  of  their  perfor¬ 
mance.  These  include  schemes  such  as  CSMA,  random  polling,  hierarchical  access  schemes 
(which  use  random  access  in  the  lightly  loaded  portion  of  a  netwOTk  and  reserved  bandwidth  in 
the  more  heavily  used  portions  of  a  network),  multiaccess  in  bus  netwmics  which  approach  giga¬ 
bit  speeds  (thereby  changing  the  latency  vs.  bandwidth  bottleneck  tradeoff),  an  analysis  of  ac¬ 
cess  systems  under  dynamically  changing  loads,  and  the  effect  of  the  overiiead  of  switching  in 
some  of  these  multiaccess  systems.  The  general  flavOT  here  was  to  provide  more  effective  ana¬ 
lytic  tools  to  be  able  to  predict  the  performance  of  a  variety  of  multiaccess  schemes  in  a  number 
of  different  scenarios. 
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We  have  also  looked  at  distributed  database  systems  using  what  is  called  optimistic  concurrency 
control;  specifically  we  are  concerned  with  the  perfminance  of  such  systems  as  compared  to  the 
conservative  locking  schemes  which  are  used  in  the  more  classical  database  control  systems. 
We  provided  an  analysis  and  simulation  of  optimistic  concurrency  control  schemes  and 
identified  their  effectiveness  in  a  variety  of  different  configurations.  Lastly,  in  the  area  of  distri¬ 
buted  systems  we  developed  and  analyzed  a  tree  search  algorithm  using  parallel  processors. 

The  third  major  area  of  our  research  involved  distributed  communications.  This  is  a  very  rapidly 
growing  area  as  we  move  into  the  domain  of  gigabit  per  second  networks  and  the  paper  by 
Kleinrock  entided  "ISDN  -  The  Path  to  Broadband  Networks"  is  included  in  the  appendix  of  this 
report  as  a  summary  of  some  of  out  thinking  and  findings  in  this  area.  We  are  fimher  looking 
into  the  details  of  congestion  control  in  the  LAN-to-LAN  interconnect  problem,  and  have  made 
some  preliminary  studies  of  wavelength  division  muldplexing  in  optical  networks.  This  discus¬ 
sion  of  wavelength  division  multiplexing  has  also  been  extended  to  the  area  of  metropolitan  area 
networks. 

Overall  our  research  progress  has  advanced  in  these  many  domains  with  considerable  success  in 
providing  an  understanding  of  the  underlying  behavior  of  parallel  and  distributed  systems.  We 
have  created  a  number  of  evaluation  tools  for  understanding  these  systems,  we  have  studied  the 
ways  in  which  the  algorithms  and  architectures  behave  with  respect  to  each  other  and  their  im¬ 
pact  upon  each  other,  we  have  developed  load  balancing  techniques  and  distributed  database 
techniques  along  with  their  analyses,  and  have  in  general  advanced  the  state  of  the  art  in  order  to 
provide  a  variety  of  tools,  techniques,  experimental  measurements  and  implementations  for  the 
understanding  of  the  behavior  of  parallel  systems. 
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4.  Baker,  Rush  "Distributed  Simulation  in  the  BBL  Multicomputer  System"  January  1989. 
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Abstract 

Distributed  systems  frequently  have  large 
numbers  of  idle  computers  and  workstations. 

If  these  could  be  harnessed,  then  consider¬ 
able  computing  power  would  be  available  at 
low  cost.  We  analyze  such  systems  using  a 
very  simple  model  of  a  distributed  program 
(a  fixed  amount  of  work)  to  see  how  the  use 
of  transient  processors  affects  the  program’s 
service  time. 

1  Networks  of  Transient 
Processors. 

Networks  of  computers  are  fairly  common  in  busi¬ 
ness  and  research  environments  throughout  the  world. 
Originally  motivated  by  a  desire  to  ease  data  and  de¬ 
vice  sharing,  many  networks  have  grown  in  speed  and 
sophistication  to  the  point  that  distributed  processing 
can  be  performed  on  them.  These  networks  vary  in  size 
from  a  handful  of  personal  computers  on  a  low-speed 
network,  to  thousands  of  workstations  and  a  variety  at 
larger  machines  on  a  high-speed,  fiber-optic  network. 
A  typical  example  is  that  of  workstations  on  a  high¬ 
speed,  local  area  network  in  a  research  laboratory.  Not 
only  are  there  many  machines,  well  ccmnected  by  the 
network,  but  the  users  are  likely  to  demand  more  and 
more  computing  power. 

On  these  networks,  we  often  have  the  situation  that 
many  ci  the  personal  computers  and  workstations  are 
sitting  idle,  waiting  for  their  users,  and  thus  being 
wasted.  If  we  could  recover  this  wasted  time  for  usefiil 
processing,  then  we  would  have  considerable  comput¬ 
ing  power  available  to  us  at  low  cost  (6).  We  refer  to 
these  processors,  which  are  sometimes  busy  and  some¬ 
times  not,  as  irantieni  processors. 

This  situation  has  analogies  to  time-sharing.  In  the 
past,  institutions  had  one  large  resource,  a  m^nframe 
computer,  that  was  shared  by  many  users.  As  net¬ 
works  of  workstaticNM  develop,  the  largest  computing 
resource  is  no  longer  a  single  machine,  but  is  instead 
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Research  Projects  Agracy  under  contract  MDA  903-82- 
C0084,  Advanced  Tdeprocessing  Systems,  and  contract 
MDA  903-87-C0883,  Parallel  Systems  Laboratory. 


the  aggregate  computing  power  of  the  workstations.  In 
the  same  manner  that  it  has  been  possible  for  many 
people  to  share  a  single  machine,  by  using  the  idle  time 
of  one  person  to  run  the  program  of  another,  so  it  is 
now  possible  to  share  a  network  of  workstations  for 
large,  distributed  computations. 

Whether  this  is  technically  feasible  or  not  depends 
on  a  variety  of  factors,  such  as  the  characteristics  of 
the  communications  medium,  the  characteristics  of  the 
computers,  and  the  statistical  characteristics  of  the 
user  population.  Mutka  and  Livny  [13],  and  Nichols 
[14]  have  shown  that  under  at  least  some  circum¬ 
stances,  this  is  very  practical  and  useful. 

fVom  an  analytical  point  of  view,  we  would  like  to 
have  a  queueing  model  of  a  netwwk  of  transient  pro- 
ces8<»8  executing  distributed  programs.  In  this  paper 
we  take  a  first  step  toward  this  end  by  analyzing  the 
service  time  for  a  very  simple  model  of  a  distributed 
program  (a  fixed  amount  of  work)  to  see  how  the  use 
of  transient  processors  affects  what  would  otherwise 
be  a  deterministic  service  process. 

2  The  Model. 

Assume  that  we  have  a  network  of  M  identical  proces¬ 
sors,  and  we  wish  to  run  a  program  that  will  require 
a  total  of  W  seconds  of  work.  In  general,  a  program 
consists  at  multiple  stages  of  wwk,  each  of  which  must 
be  completed  before  the  start  of  the  next.  For  this 
paper  we  assume  that  the  program  has  only  one  stage 
of  work,  and  furthermore,  we  assume  that  the  work  in 
any  stai^  is  infinitely  divisible,  and  therefore  is  always 
spread  evenly  among  the  available  processors. 

Each  processor  has  a  capacity  of  one  second  of  work 
per  second.  A  processor  alternates  between  a  non- 
avatfak/e  period,  when  someone  is  using  it,  and  an 
avat/ak/e  period,  when  it  is  sitting  idle.  We  assume 
that  the  length  of  non-available  periods  is  randomly 
distributed  with  mean  f„  and  variance  and  that 
the  length  of  available  periods  is  also  randomly  dis¬ 
tributed  from  a  (possibly  different)  distribution  with 
mean  t«  and  variance  al.  We  wish  to  run  our  program 
on  the  network  of  processors,  using  processors  while 
they  are  in  their  available  periods.  The  finishinj  time 
of  the  program  is  given  by  a  random  variable  /,  with 
probabiUty  distribution  /(f)  ,  average  J,  and  Laplace 
transform  F*(s).  The  purpose  of  this  p^ier  is  to  find 
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f(t)  ,  or,  in  broader  terms,  to  examine  the  potential 
use  of  currently  wasted  cycles.  To  do  so,  we  examine  a 
related  function,  u/(u|t),  the  probability  density  func¬ 
tion  of  the  amount  of  work,  u,  completed  by  time  t, 
which  has  mean  Wl  and  variance  Var(W'i].  Then,  for 
a  given  W,  f(t)  is  the  distribution  of  the  time  for  the 
completed  work  to  first  reach  W,  a  point  also  known 
as  the  first  passage  time. 

If  the  amount  of  work  that  we  wish  to  do  is  small, 
relative  to  t,  and  tn ,  then  the  finishing  time  is  highly 
dependent  upon  the  states  of  the  processors  when  the 
program  arrives.  If,  for  example,  we  have  a  small 
amount  of  work  to  do  on  a  single  processor,  either 
the  processor  is  available  with  no  delay,  or  the  pro¬ 
cessor  is  non-available,  and  we  must  wait,  on  average, 
t„  seconds  (assuming  exponentially  distributed  non- 
available  periods)  before  we  can  even  start  the  work. 
If  t„  is  not  small  relative  to  W,  then  state  of  the  pro¬ 
cessor  when  the  program  arrives  strongly  affects  the 
finishing  time  distribution.  In  this  paper,  we  use  two 
techniques  to  mitigate  the  effect  of  this  on  the  anal¬ 
ysis.  The  single  processor  models  make  assumptions 
about  the  time  that  the  program  arrives  (either  in 
available  period  or  at  the  beginning  of  a  non-available 
period).  The  multiprocessor  model  assumes  that  W  is 
large  relative  to  ta  and  tn ,  so  the  effect  of  initial  condi¬ 
tions  is  negligible.  Work  is  underway  for  the  situations 
where  W  is  not  large. 

Our  analysis  ultimately  allows  for  arbitrary  distri¬ 
butions  of  the  lengths  of  available  and  non-available 
periods,  but  in  our  examples,  we  assume  exponential 
distributions.  Using  the  average  available  and  non- 
available  times  measured  by  Mutka  and  Livny,  the  ex¬ 
amples  in  this  paper  are  generally  based  on  the  follow¬ 
ing  parameters;  ta  =  91  minutes,  („  =  31.305  minutes, 
W  =  10^  minutes  for  single  processor  examples,  and 
W  =  10^  minutes  for  multiprocessor  [M  =  100)  ex¬ 
amples.  We  choose  W  large  relative  to  ta  and  t„  for 
the  reasons  mentioned  in  the  previous  paragraph. 

We  ignore  communications  overhead  and  task  prece¬ 
dence  issues  in  this  paper,  and  assume  that  our  pro¬ 
gram  can  use  all  available  processors  at  any  given  time. 
The  results  of  this  paper,  then,  provide  upper  (i.e.  op¬ 
timistic)  bounds  on  the  best  performance  achievable 
in  this  situation. 

In  the  remainder  of  this  paper,  we  first  discuss  pre¬ 
vious  work  by  others,  then  analyze  the  problem  for  1 
processor.  We  find  expressions  for  u;(u  1 1)  and  /(<) 
in  the  single  processor  case.  We  then  use  u;(u  \  t)  to 
find  /(f)  for  M  processors  and  compare  it  to  the  single 
processor  catse. 

3  Previous  Work. 

A  single  transient  processor  can  be  modeled  in  a  va¬ 
riety  of  ways;  as  a  priority  queue,  as  a  queue  with 
vacations,  as  an  unreliable  system,  or  as  a  cumula¬ 
tive,  alternating  renewal  process.  Of  these  methods, 


we  choose  that  last  because  it  provides  the  asymptotic 
distribution  of  w(u  |  f )  in  a  very  simple  form  and  for 
arbitrary  available  and  non-avidlable  period  distribu¬ 
tions.  We  now  briefly  discuss  the  other  alternatives. 

3.1  Preemptive  Priority  Queues. 

We  can  model  a  transient  processor  using  a  preemptive 
priority  queueing  system  with  two  classes.  The  high 
priority  class  represents  non-available  periods  that  in¬ 
terrupts  the  service  of  distributed  programs,  repre¬ 
sented  by  the  jobs  in  the  low  priority  class.  Such  sys¬ 
tems  do  not  completely  model  a  transient  processor 
because  high  priority  jobs  continue  to  arrive  while  a 
high  priority  job  is  in  service,  and  this  represents  a 
queueing  of  non-available  periods.  If  we  do  not  object 
to  this,  and  if  we  assume  that  both  available  and  non- 
available  periods  arrivals  are  from  (different)  Poisson 
processes,  then  the  Laplace  transform  of  the  “comple¬ 
tion  time”  (our  /’*(«))  is  known  [8],  and  from  that  we 
can  get  its  mean  and  variance. 

3.2  Queues  with  Vacation. 

Another  model  of  a  transient  processor  is  a  queueing 
system  in  which  the  server  goes  on  vacations.  In  par¬ 
ticular,  we  require  that  the  vacations  occur  randomly 
and  preempt  any  customer  in  service.  If  the  available 
periods  are  exponentially  distributed,  Gaver  [2]  pro¬ 
vides  an  analysis.  For  non-exponential  available  pe¬ 
riods,  Federgruen  and  Green  [5]  have  analyzed  such 
queues. 

3.3  Unreliable  Systems. 

Reliability  analysis  concerns  itself  with  the  availability 
of  a  system  over  time,  and  some  work  has  been  done  to 
find  the  distribution  of  cumulative  av2tilability  (the  cu¬ 
mulative  amount  that  a  system  is  available  over  time). 
This  corresponds  exactly  to  accumulation  of  work  in 
a  network  of  transient  processors.  For  a  system  with 
two  states  (available  and  non-available)  Donatiello  and 
Iyer  [4]  find  the  transform  of  the  cumulative  availabil¬ 
ity,  and  they  derive  a  closed-form  expression  when  the 
time  in  each  state  is  exponentially  distributed.  Using 
their  results,  we  can  find  the  moments  of  the  amount 
of  work  done  in  a  given  amount  of  time  for  any  spe¬ 
cific  available  and  non-available  period  distributions. 
However,  we  were  unable  to  derive  general  results  in 
terms  of  the  distributions’  moments,  and  therefore  we 
turned  to  the  cumulative,  alternating  renewal  analysis 
described  later  in  this  paper.  Although,  their  results 
do  not  directly  provide  the  distribution  of  the  first  pas¬ 
sage  time,  the  transform  of  this  may  be  obtained  using 
techniques  from  the  cumulative,  alternating  renewal 
analysis. 

De  Souza  e  Silva  and  Gail  [3]  discuss  the  calcula¬ 
tion  of  cumulative  availability  in  systems  which  can  be 
modeled  as  homogeneous  Markov  chains.  They  find  a 
general  method  for  calculating  cumulative  availability, 


and  further  find  good  techniques  for  numerical  evalua- 
tion  of  their  method.  The  SAVE  (System  Availability 
Estimator)  program  [7]  implements  their  method.  For 
numerical,  not  approximate  analytic,  results,  this  is  an 
excellent  approach. 


4  Results  for  One  Processor. 


We  use  two  methods  for  deriving  for  the  finishing  time 
of  a  program  on  one  processor.  The  first  analysis  pro¬ 
vides  the  distribution  of  the  first  passage  time  with 
some  restrictions  on  the  distributions  of  the  available 
and  non-available  periods  (see  section  4.1);  however,  it 
does  not  yield  the  distribution  of  the  amount  of  work 
done  over  time.  The  second  analysis,  using  a  cumu¬ 
lative,  2dternating  renewal  process,  provides  us  with 
the  distribution  of  w{u  1 1)  for  arbitrary  available  and 
non-available  period  distributions,  but  only  with  the 
transform  of  the  first  passage  time  distribution.  Given 
w(u|t)  from  the  second  atnalysis,  we  can  then  develop 
a  model  for  M  processors. 

We  may  easily  derive  the  averages  Wi  and  J.  In  a 
long  period  of  time,  the  processor  spends,  on  average, 
a  fraction  ta/ita  +  tn)  of  its  time  in  available  periods. 
Thus  in  t  seconds,  the  average  amount  of  work  com¬ 
pleted  is  t  times  this  fraction,  or 


w:= 


ia  "I" 


t. 


(1) 


Because  the  processor  completes,  on  average,  ta/{ta  + 
tn)  seconds  of  work  per  second,  the  reciprocal  of  this  is 
the  number  of  seconds  it  takes  to  complete  one  second 
of  work.  Multiplying  this  by  W  we  find  the  average 
first  passage  time. 
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If  we  wish  to  account  for  multiple  processors,  the  aver¬ 
age  amount  of  work  done  per  time  period  is  increased 
by  a  factor  of  M,  the  number  of  processors,  and  the 
first  passage  time  is  likewise  decreased  by  a  factor  of 
M,  giving  us: 


w;= 

Mt. 

(3) 

ta  +  tn 

7= 

ta 

Mi, 

W 

(4) 

As  we  will  find,  this  simple  analysis  accurately  charac¬ 
terizes  the  system  when  our  program  takes  a  long  time 
relative  to  the  average  length  the  available  and  non- 
available  periods. 


4.1  Direct  Analysis. 

Assume,  for  the  moment,  that  we  have  only  one  proces¬ 
sor  (Af  =  1).  If  our  program  starts  when  the  processor 
is  available,  as  shown  in  Figure  1,  it  wiU  finish  at  time 
W  +  w,  where  w  is  the  addition^  (wasted)  time  the 
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Figure  1:  Time  for  one  processor  to  finish  W  seconds 
of  work. 


program  spends  in  the  system  because  the  processor 
is  sometimes  in  non-available  periods. 

Because  the  non-available  periods  are  independent 
and  identically  distributed  random  variables  with  ex¬ 
ponential  distributions,  the  probability  distri¬ 

bution  function  of  the  additional  time  in  non-available 
periods,  given  that  we  have  t  such  periods,  is  an  Erlang 
distribution;  that  is 


^  (lAn)(fAn)-^ 
dt  (i -  1)! 


The  number  of  non-available  periods  that  "arrive”  dur¬ 
ing  W  seconds  has  a  Poisson  distribution  with  rate 
1/tg.  Thus,  unconditioning  on  this  number,  we  get 
that  the  density  of  ti;  is 

dt 

e"  «o(0 

ift>0 


(6) 

ift  =  0 


where  uo(0  is  a  unit  impulse  at  t  =  0.  For  t  >  0,  this 
may  be  further  reduced  to 


dP*{t) 

dt 


where  It(z)  is  the  modified  Bessel  function  of  the  first 
kind  of  order  1.  Figure  2  shows  the  distribution  of  the 
first  passage  time  using  t,  =  91  and  tn  =  31.305,  and 
10*. 

To  find  the  mean  and  variance  of  this  distribution, 
it  simplifies  the  analysis  to  model  the  sequence  of 
non-available  periods  using  a  series-parallel  queueing 
server,  shown  in  Figure  3,  and  then  use  known  results 
for  such  servers.  In  this  model,  each  of  the  infinite 
number  of  sub-servers  (the  numbered  circles  in  the  fig¬ 
ure)  has  the  same  distribution  as  a  single  n(»-available 
period,  and,  in  fact,  represents  a  non-available  period. 
We  adjust  the  transition  probabilities  pi  so  that  the 
nun^er  of  sub-servers  a  program  passes  through  has 
the  same  distribution  as  the  number  of  non-available 
periods  arriving  in  IV  seconds.  Upon  entering  the 
series-parallel  server,  the  job  imme^ately  leaves  the 
server  with  probability  po  ,  or  the  job  enters  sub-server 


Figure  2:  First  Passage  Time  Density  for  Direct  Anal¬ 
ysis. 


where  Vxi*)  =  is  the  s-transfnrm  of  a  Poisson 

distribution  with  parameter  A. 

We  multiply  this  by  to  account  for  the  W 

seconds  that  we  are  required  to  work,  and  finally  find 
the  transform  for  the  distribution  of  time  to  finish  W 
seconds  of  work,  i.e., 

F*(s)  =  (11) 


Taking  derivatives,  we  find  that  the  mean  time  re¬ 
quired  to  finish  W  seconds  cS  work  is 
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(12) 


and  the  variance  of  this  time  is 


(13) 


’•ft. 
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Figure  3:  Series-parallel  model  of  extra  time  needed 
to  complete  W  seconds  of  work. 


1  with  probability  1  —  po-  Then  each  time  a  job  leaves 
a  sub-server  i,  a  similar  choice  is  made  between  leav¬ 
ing  the  series-parallel  server  (with  probability  Pi)  or 
continuing  on  to  the  next  sulnserver  (with  probabil¬ 
ity  1  —  Pi).  For  our  purposes,  we  adjust  the  prob¬ 
abilities  Pi  so  that  the  number  of  sub-servers  passed 
through  has  a  Poisson  distribution.  To  compute  these 
Pi,  we  note  first  that  po  =  For  notational  con¬ 

venience,  let  Vx{i)  =  7r«“*i  the  i**  term  of  a  Pois¬ 
son  distribution  with  parameter  A.  Next,  we  see  that 
(1  -  po)pi  =  ?^w/«,(l),  and  we  immediately  have  that 


l-'Piv/i.(0)- 
It  may  be  proved  by  induction  that 


^  -  EJ=o^w'/«.(;) 


(8) 


Using  these  Pi’s,  and  the  expression  in  Kleinrock  [10] 
for  W*(s),  the  Laplace  transform  of  the  density  of  the 
time  required  to  pass  through  a  series-parallel  server, 
we  get 


which,  after  some  manipulation,  becomes 


Note  that  Equation  12  agrees  with  Equation  2.  This 
mean  and  variance  may  also  be  derived  by  viewing  the 
distributi(Mi  as  a  constant  plus  a  random  (Poisson- 
distributed)  sum  of  Erlang  random  variables,  and  us¬ 
ing  the  well-known  formulas  for  the  mean  and  variance 
of  a  random  sum  of  random  variables  [10]. 

We  can  modify  this  analysis  for  non-exponential 
non-available  time  distributions,  but  it  still  requires 
exponential  available  times  so  that  the  number  of  non- 
av^able  periods  in  W  seconds  has  a  Poisson  distribu¬ 
tion.  If  we  wish  to  allow  arbitrary  available  period 
distributions  as  well,  then  we  need  a  more  sophisti¬ 
cated  method  of  counting  non-available  periods,  and 
this  leads  to  analyzing  the  situation  as  a  renewal  pro¬ 
cess. 


4.2  Analysis  as  a  Cumulative, 

Alternating  Renewal  Process 

To  allow  arbitrary  distributions  for  the  available  and 
non-available  periods,  we  follow  the  presentation  of 
Cox  [1]  in  his  treatment  of  cumulative,  alternating  re¬ 
newal  processes.  As  before,  the  single  processor  has 
a  capacity  of  one  second  of  work  per  second,  and  we 
wish  to  find  the  distribution  of  the  time  required  for 
this  processor  to  finish  W  seconds  of  work.  Let  X-  be 
the  duration  of  the  i***  non-available  period,  and  X" 
be  the  duration  of  the  available  period,  as  shown 
in  Figure  4.  Let  the  renewal  points  for  our  alternating 
renewed  process  be  the  beginning  of  the  non-available 
periods,  shown  as  heavy  dots  in  the  figure;  the  time  be¬ 
tween  renewal  points  is  then  Xi  —  X-  +  X".  This  has 
mean  E[A]  =  tg+t„  and  variance  Var[X]  =  q-  (r*. 
We  assume  that  t  =  0  occurs  at  the  beginning  of  a 
renewal  period. 

To  form  a  cumulative  process  from  this,  define  Wi 
to  be  the  amount  of  work  completed  in  each  renewal 
period:  Wi  =  X",  with  mean  ia  and  variance  Let 
Zt  be  the  sum  of  all  the  available  time  up  to  time  t, 
excepting  that  in  the  current  available  period,  if  the 
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Figure  4:  Cumulative  Renewal  Process 


process  is  in  such  a  period  at  time  t. 


N, 

Zt  =  W  =  l,2,...)  (14) 

1=1 

=  0  iNt  =  0) 


where  Nt  is  the  number  of  renewals  in  (0,t].  Asymp¬ 
totically,  Zt  has  the  same  properties  as  w(u  1 1),  the 
process  that  accumulates  the  true  total  available  time 
up  to  time  t;  however,  Zt's  analysis  is  more  tractable 
than  that  of  u;(u|t). 

Cox’s  analysis  dlows  the  available  and  non-available 
periods  to  have  arbitrary  distributions.  For  t  large,  Zt 
is  the  sum  of  many  independent  random  variables,  and 
it  is  asymptotically  normal  with  mean 

and  variance 


Var[Zt]  = 


(ta  + 


(16) 


Using  exponentially  distributed  available  and  non- 
available  periods,  these  become; 


E[Zt]  = 


"b 


t 


(17) 


As  noted  before,  the  asymptotic  properties  of  Zt  are 
identical  to  the  asymptotic  properties  of  w(u|t). 

Comparing  this  to  the  result  of  the  direct  analysis, 
we  note  that  for  t  equal  to  the  mean  first  passage  time 
(Eqn.  12),  we  have  done,  on  average,  W  seconds  of 
work,  as  we  expected. 

In  his  derivation,  Cox,  assuming  that  Wt  is  inde¬ 
pendent  of  X"  for  all  i,  derives  a  double  transform  for 
/(t) .  Unfortunately,  this  transform  is  very  difficult  to 
invert,  but  the  asymptotic  distribution  of  Zt  is  really 
what  we  need  so  we  can  use  it  in  the  next  analysis. 


5  Results  for  M  Processors 

Because  the  amount  ci  work  done  by  one  transient 
processor  is  the  sum  of  a  (possibly  large)  number 
available  periods,  the  tot^  work  done  in  time  t  is 
asymptotically  normal  with  mean  and  variance  given 
in  Equations  17  and  18,  respectively.  If  1.  <  and 


m 


Figure  5:  First  Passage  Time  Distributions  for  Direct 
and  Brownian  Motion  Analyses. 


Figure  6;  First  Passage  Time  Densities  for  Brownian 
Motion  Model  for  Various  t„. 


tn  W,  then  it  is  reasonable  to  use  Brownian  motion 
as  a  model  of  an  M  processor  system.  All  M  proces¬ 
sors  are  assumed  to  be  independent,  so  the  amount  of 
work  done  by  time  t  is  the  sum  of  M  independent,  (ap¬ 
proximately)  normally  distributed  random  variables, 
with  mean 


Ft  = 


fo  "b  ffi 


Mt  =  PaMt 


and  variance 


+  vltl) 


Mt. 


(19) 


(20) 


For  exponential  available  and  non-available  distribu¬ 
tions,  Eqn.  20  becomes: 


2(1.1.)’  ...  2ri(l-P.)Mt 

*'  -  -  — z 


where  p,  =  t,/{t,  + 1„). 

With  these  as  the  parameters  for  our  Brownian  mo¬ 
tion,  the  density  of  the  time,  f,  that  it  takes  for  M 


»V>10000 


(minutes) 


Figure  7;  First  Passage  Time  Density  for  Brownian  Figure  8:  ojfJ  with  various  and  t„. 

Motion  Analysis. 


processors  to  finish  W  seconds  of  work  is  well-known 
(e.g.  see  Karlin  and  Taylor  [9])  and  is  given  by: 


m  = 


W 


exp 


[—W~\ 


(22) 


This  has  mean 


and  variance 


,  W  ty(t.-Kn) 
^  ~  I  ~  M  t« 


3 


TF 


(23) 


(24) 


and  for  exponential  distributions,  the  latter  beconKS 

(25) 


>  M2  U  ■ 


Equation  22  is  the  main  result  of  this  paper.  Note  that 
it  makes  no  assumptions  about  the  distributions  of  the 
available  and  non-available  periods,  except  that  their 
variances  are  finite,  and  only  the  distributions’  means 
and  variances  appear  in  the  first  passage  time. 

Note  that  for  the  case  M  =  1,  this  mean  and  vari¬ 
ance  agree  with  our  single  processor  analysis  of  section 
4.1.  The  first  passage  time  densities  for  both  the  sin¬ 
gle  processor  analysis  and  the  multiprocessor  analysis 
with  M  =  1  are  shown  in  Figure  5. 

Figure  6  shows  the  distribution  of  first  passage  time 
for  various  t„  with  ta  =  91,  M  =  100,  and  =  10^. 
Using  the  t.  =  91  and  f„  =  31.305,  our  job  of  10^ 
minutes  would  take  about  a  week  to  run  on  a  single, 
dedicated  processor.  When  run  on  a  network  of  100 
transient  processors,  it  would  take  134.06  minutes,  or 
about  2.25  hours.  The  distribution  of  this  first  passage 
time  is  shown  in  Figure  7.  Note  that  although  W  >  t, 
and  W  ><„,  we  have  that  W/M,  the  finishing  time  if 
the  processors  were  fully  dedicated  to  the  program,  is 


close  to  the  average  length  of  an  available  period,  and 
it  is  remarkable  that  the  curve  is  still  so  symmetrical. 
It  is  useful  to  examine  the  ratio  of  fff  to  J,  namely: 


7  \/W 


(26) 


to  see  what  happens  to  the  distribution  as  the  pa¬ 
rameters  change.  For  exponential  distributions  this 
becomes 


and  it  is  this  equation  that  we  examine  in  more  detail. 
Figure  8  plots  Eqn.  27  for  fixed  t„  and  W,  and  varying 
ta-  Note  that  in  this  figure,  t„  =  30  minutes  is  the 
lowest  line  and  i„  =  1500  minutes  is  the  uppermost 
line. 

Because  we  assumed  t„  W,  this  ratio  tends  to 
be  less  than  1,  reaching  a  peak  when  ta  =  in-  If 
fix  t„/W  and  let  t„/ta  go  to  infinity  (which  implies 
ta  — »  0),  the  ratio  goes  to  0.  We  explain  this  by  noting 
that  for  small  ta,  it  takes  very  many  available-non- 
available  cycles  before  the  work  is  finished.  The  law 
of  large  numbers  insures  that  the  first  passage  time 
distribution,  which  is  the  sum  of  these  many  periods, 
will  then  be  tight  about  its  mean. 

If,  on  the  other  hand,  we  let  ta  — *  oOi  the  ratio 
of  the  standard  deviation  to  the  mean  goes  to  zero 
once  again.  When  ta  is  large  relative  to  t„,  the  non- 
av«iable  periods  become  negligible,  as  if  the  processors 
are  always  available.  Again,  the  first  passage  time  dis¬ 
tribution  becomes  very  tight  about  its  mean  because 
non-available  time  periods  add  little  variability  to  the 
finishing  time. 

Given  that  the  first  passage  time  distribution  is  tight 
about  its  mean,  (i.e.  f#  >•  tn,  or  <  1„,  or  W  >  1„), 
it  may  be  accurate  enough  to  consider  the  distribu¬ 
tion  as  an  impulse  at  the  mean  finishing  time  (in  the 
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^tn  ftg 


W  l-¥t„/ta 


(27) 
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spirit  of  the  law  of  large  numbers).  Using  the  previous 
example  again,  we  find  (r^  =  21.53,  and  approximat¬ 
ing  f{t)  as  a  normal  distribution  (discussed  below), 
we  find  that  90%  of  the  time,  programs  requiring  10^ 
minutes  of  work  will  finish  within  7.6  minutes  of  the 
134.4  minute  mean  finishing  time. 

The  central-limit  theorem  says  that  f{t)  will  tend 
toward  a  normal  distribution  when  meuiy  available — 
non-available  periods  occur  before  the  program  com¬ 
pletes  (i.e.  IV  >  la  »nd  IV  >  t„).  To  approximate 
the  first  passage  time,  we  use  a  normal  distribution 
with  the  same  mean  amd  variance  as  the  first  passage 
time  distribution: 


m  = 


1 


e-(‘-7)"/(3*?) 


(28) 


When  the  mode  of  the  first  passage  time  distribution 
is  close  to  its  mean,  a  normal  distribution  well  approx¬ 
imates  the  first  passage  time.  The  mode  is: 


1 


^mode  —  2 


4W^ 

\l  t 

'W 

(29) 


Comparing  this  to  the  mean,  we  find  that  the  per¬ 
centage  difference  between  the  two  is  approximately 
Zai/2bW,  which,  as  we  would  expect,  shrinks  as  W 
grows. 


6  Conclusion 


We  have  analyzed  a  network  of  transient  processors, 
and  determined  the  probability  density  of  the  length 
of  time  it  takes  to  finish  a  fixed  amount  of  work. 
The  main  result  for  an  M  processor  network  is  given 
in  Equation  22,  and  it  is  valid  for  general  available 
and  non-available  period  distributions.  Simulations 
confirm  that  Brownian-motion-with-drift  is  an  accu¬ 
rate  model  of  system  performance  under  the  assump¬ 
tions  given  above.  With  large  programs  that  run  for 
a  long  time  relative  to  the  length  of  available  and 
non-available  periods,  the  central  limit-theorem  ap¬ 
plies,  and  the  Brownian-motion-with-drift  model  re¬ 
mains  good  regardless  of  the  distributions  of  the  avail¬ 
able  and  the  non-available  periods.  Under  these  as¬ 
sumptions,  the  distributbn  of  finishing  time  will  be 
very  tight  about  its  mean,  and  is  well  approximated 
by  a  normal  distribution. 

It  does  remain  to  account  for  communication  over¬ 
head  and  precedence  relationships,  but  it  is  likely  that 
these  can  be  accommodated,  or  at  least  approximated, 
within  the  model. 

The  analysis  in  this  paper  has  not  examined  the  ef¬ 
fect  of  multiple  programs  in  the  network.  We  may 
now  use  the  first  passage  time  distribution  (Eqn.  22), 
as  the  service  time  in  a  queueing  system  that  repre¬ 
sents  the  network.  If  each  job  gets  the  whole  network 
and  they  must  queue,  then  a  G/G/1  queue  is  a  good 
model.  If  all  the  programs  in  the  network  share  the 


processors  equally,  then  we  could  model  the  network 
as  an  M/G/l  processor-sharing  system.  The  analysis 
of  such  systems  remains  for  a  future  paper. 

It  is  well  known  that,  for  a  given  total  processing  car 
pacity,  the  average  response  time  is  shortest  if  we  use 
one  large  processor  rather  than  many  small  processors 
[11].  From  this  perspective,  the  trend  toward  individ¬ 
ual  workstations  is  a  curious  one.  However,  this  result 
assumes  that  each  program  executes  on  only  one  pro¬ 
cessor.  If  we  distribute  the  program  over  all  the  small 
processors,  then  we  may  recover,  at  least  partially,  the 
response  time  advantages  of  a  large,  central  system, 
while  retaining  the  advantages  of  individual  workstar 
tions. 
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Abstract — We  model  a  Job  ia  a  paraDd  pioccsalag  system  as 
a  sequence  of  stages,  each  of  whM  reqairn  a  certaia  integral 
number  of  processors  fbr  a  certain  interval  of  time.  With  this 
model  we  derive  the  speednp  of  the  system  for  two  cases:  systems 
with  no  arrivals,  and  systems  with  arrivals.  In  the  case  with 
no  arrivals,  our  speednp  icsalt  is  a  generaUzmion  of  Amdahl’s 
Law.  We  extend  the  notion  of  ’’power”  (the  simplest  delnition 
is  power  =  throughput/response  time)  as  previousiy  appHed  to 
general  queueing  and  computer<ommnnlcatton  systems  to  our 
case  of  parallel  processing  systems.  With  tids  delaltioa  of  power 
we  are  able  to  had  dm  optimal  system  operatfaig  point  (ik,  the 
opttanal  input  rate  of  Johs)  and  the  optimal  number  of  processors 
to  use  in  the  paraHei  processing  system  such  that  power  is 
maximised.  Many  of  the  results  ^  the  case  of  arrivals  are  the 
same  as  for  the  case  of  no  anrivais.  A  tunillar  and  latnltiveiy 
pleasiag  result  b  obtained,  which  states  that  the  average  number 
of  Jobs  in  the  system  with  arrivals  equals  unity  when  power  is 
maximised. 

We  also  Bsodei  a  Job  in  a  way  such  that  the  number  of  proces¬ 
sors  ret^ired  is  a  contianons  variable  that  changes  coatinnonsly 
over  tisM.  The  same  perfonnaace  indices  and  parameters  studied 
In  the  discrete  model  are  evalnated  for  this  contianons  model. 
These  continuous  resutts  are  more  easily  obtained,  are  easier  to 
state,  and  are  simpler  to  interpret  than  for  the  discrete  model. 

Index  Terms — Amdahl’s  Law,  multiprocessing,  optimal  design, 
parallel  processing,  power,  processor  eflkiency,  sp^np,  system 
ntiUiatioa. 


I.  Introduction 

AS  parallel  computing  systems  proliferate,  the  need  for 
effective  performance  evaluation  techniques  becomes 
ever  more  important.  In  this  paper,  we  study  certain  funda¬ 
mental  performance  indices,  namely,  speedup,  response  time, 
efficiency,  and  power,  and  solve  for  the  optimal  operating  point 
of  these  systems.  Specifically,  by  maximizing  “power,”  we  are 
able  to  find  the  optimal  input  rate  of  jobs  and  the  optimal 
number  of  processors  to  use,  given  a  characterization  of  the 
workload. 

We  model  a  parallel  processing  system  as  a  system  with  a 
single  queue  of  waiting  jobs.  Our  first  model  (in  Section  IV) 
assumes  that  only  a  single  job  needs  to  be  processed  Our 
second  model  (in  Section  V)  allows  a  stream  of  arrivals  to 
enter  the  system;  however,  only  one  job  may  be  admitted 
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Fig.  I.  Job  profile,  (a)  Unlimited  number  of  processors.  ( b)  Limited  number 
of  processors  {P  =  A). 

into  service  at  a  time,  following  a  FCFS  discipline,  while 
the  others  wail  in  the  queue.  Both  models  deal  with  jobs  as 
follows.  While  in  service,  the  system  provides  a  maximum  of 
P  parallel  processors  to  work  on  the  job.  A  job  is  modeled 
as  a  sequence  of  independent  stages  which  must  be  processed, 
where  the  number  of  processors  desired  by  the  job  in  each 
stage  may  be  different.  If.  for  some  stage,  the  job  in  service 
requires  fewer  processors  than  the  system  prrxvides.  then  the 
job  will  use  aft  that  it  needs  and  the  other  processors  will  be 
idle  for  that  stage.  If.  for  some  other  stage,  the  job  in  service 
requires  more  processors  than  the  system  provides,  then  it  w  ill 
use  all  the  processors  in  the  system  (in  a  prtKessor  sharing 
fashion  (10))  for  an  extended  period  of  time  such  that  the 
total  work  served  in  that  stage  is  conserved.  An  example  is 
given  in  Fig.  1  in  which  the  total  processing  work  required  by 
a  job  is  U  =  ‘21  s.  In  this  example,  if  /’  >  (i.  then  it  takes 
8  s  to  complete  the  job  as  shown  in  Fig.  1(a).  whereas  if  only 
P  =  1  processors  are  provided,  then  it  takes  9  s  as  shown  in 
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Fig-  1(b),  in  which  case  12  s  of  processor  capacity  are  wasted. 

The  model  described  above  has  been  highly  idealized. 
In  particular,  we  are  neglecting  some  of  the  following  im¬ 
portant  aspects  of  the  workload.  First,  we  do  not  allow 
general  precedence  relations  among  the  ta.sks.  Our  precedence 
structure  is  equivalent  to  a  series-parallel  task  graph  with 
deterministic  task  service  times  (see  [6]  for  the  definition  of 
the  task  graph  model  of  computation).  Second,  we  do  not 
separately  model  the  communication  times  between  tasks  (i.e., 
the  interprocess  communication  overhead).  We  hasten  to  point 
out  that  incorporating  this  overhead  is  not  simply  a  matter  of 
adding  additional  time  to  each  task's  processing  time,  since 
such  overhead  only  occurs  when  a  task  on  one  processor  must 
pass  its  results  to  a  task  on  a  different  processor;  thus  to 
properly  include  interprocess  communication  costs,  one  must 
model  the  way  in  which  tasks  are  assigned  to  processors  (i.e., 
the  task  partitioning  problem),  an  assignment  that  we  choose  to 
neglect.  Third,  we  ignore  I/O  communication  overhead  related 
to  the  management  and  execution  of  parallel  programs.  Lastly, 
we  assume  that  the  program  structure  is  infinitely  divisible, 
in  that  the  time  to  execute  w  units  of  work  is  equal  to 
max  (i<  /P,  u  /P*),  wher^  P  is  the  number  of  processors  that 
the  system  provides  for  execution  of  this  work,  and  P'  is  the 
maximum  number  of  processors  that  the  program  is  able  to 
use  for  this  work  (i.e.,  the  parallelism  for  this  work).  These 
assumptions  simplify  our  analysis  and  lead  to  idealized  results. 

Our  workload  model  was  first  reported  by  us  in  (8).  Later, 
Gelenbe  (6)  described  a  very  similar  model,  as  did  Sevcik 
[15].  Gelenbe  extended  his  model,  which  he  referred  to  as  the 
“Activity  Set  Model,"  to  include  the  effect  of  inefficient  use  of 
processors,  imbalance  of  the  workload  among  the  processors, 
and  interprocess  communication  times.  Sevcik  also  described 
ways  in  which  this  idealized  model  could  be  extended  to 
include  the  effect  of  I/O  communications,  overhead,  and 
dependencies  among  parallel  threads  assigned  to  different 
processors. 

For  such  a  parallel  processing  system  there  are  two  perfor¬ 
mance  measures  which  compete  with  each  other:  processor 
efficiency  and  mean  response  lime.  One  can  increase  the 
processor  efficiency  of  the  system  (by  reducing  the  number 
of  processors),  but  then  the  mean  response  time  will  also 
be  inaeased.  Similarly,  one  can  lower  the  mean  response 
time  (by  increasing  the  number  of  processors),  but  then  the 
processor  efficiency  of  the  system  will  also  be  lowered.  In 
this  paper  these  two  performance  measures  are  combined  into 
a  single  measure,  known  as  power,  which  increases  by  either 
lowering  the  mean  response  time  or  by  raising  the  processor 
efficiency  of  the  system.  We  seek  to  find  that  number  of 
processors  which  maximizes  power. 

Power,  studied  in  [S],  [11],  and  [12],  was  defined  for  a 
general  queueing  system  in  [12]  as 

P 

r/y 

where  p  is  defined  as  the  system  utilization,  T  is  defined  as  the 
mean  response  time,  and  z  is  defined  as  the  average  service 
time.  With  this  measure  we  see  that  an  increase  in  system 
utilization  (p)  or  a  decrease  in  response  time  (T)  increases 


the  power.  (Note  that  this  normalized  definition  is  such  that 
since  0  <  p  <  I,  and  since  1  <  T/z,  then  0  <  power  <  1.) 
The  symbol  will  be  used  throughout  to  denote  variables 
which  are  optimized  with  respect  to  power.  In  [  12]  it  was  found 
that  fo2_any  M/G/1  queueing  system  [9],  power  is  maximized 
when  .V  =1,  where  .V  =  the  average  number  of  jobs  in  the 
system.  This  result  says  that  an  M/G/1  system  has  maximum 
power  when  on  the  average  there  is  only  one  job  in  the  system. 
This  result  is  intuitively  pleasing,  since  it  corresponds  to  our 
deterministic  reasoning  that  the  proper  operating  point  for  a 
single-server  system  is  exactly  when  only  one  job  is  being 
served  in  the  system  and  no  others  are  waiting  for  service  at 
the  same  time.  In  this  paper,  our  results  also  show  that  .V  =  I 
when  power  is  maximized  with  respect  to  the  job  arrival  rate 
(A). 

One  might  argue  that  power,  as  here  defined,  is  an  arbitrary 
performance  measure.  In  response  to  this  argument  we  point 
out  that  one  can  generalize  the  definition  of  power  in  a  way 
which  allows  the  reader  to  emphasize  delay  (or  efficiency)  in 
a  variety  of  ways  so  as  to  match  his  or  her  needs.  This  issue 
is  discussed  below  in  Section  II  as  well  as  in  [5]  and  [12], 
Moreover,  other  researchers  have  seen  fit  to  optimize  power 
for  models  similar  to  ours  (see,  for  example  [4]).  An  extensive 
study  of  power  applied  to  computer  networks  is  given  m  [S]. 

An  alternative,  and  much  more  familiar,  performance  mea¬ 
sure  for  parallel  processing  systems  is  speedup,  which  de¬ 
scribes  how  much  faster  a  job  can  be  processed  using  multiple 
processors,  as  compared  to  using  a  single  processor.  Specifi¬ 
cally,  speedup  is  the  ratio  of  the  mean  response  time  of  a  job 
processed  by  a  single  processor  to  that  of  a  job  executed  in  a 
parallel  processing  system  with,  say,  P  processors.  Speedup 
and  power  are  related  and  we  discuss  how  they  interact 
throughout  this  paper.  Eager  et  al.  [4]  also  discuss  issues 
similar  to  chose  in  this  paper.  Their  focus  is  on  estimating 
speedup  and  efficiency  (for  the  no  arrivals  case  only)  simply 
from  the  value  of  the  “average  parallelism,”  which  is  defined 
as  W,  the  total  processing  work  required  by  a  job,  divided 
by  the  time  it  would  take  to  service  the  job  if  there  were  an 
unlimited  number  of  processors  available;  in  Fig.  1(a)  we  have 
W  =  24,  and  service  time  =  8,  giving  an  average  parallelism 
equal  to  3.  They  also  use  the  definition  of  power  as  we  had 
defined  in  [1 1]  and  [12]  and  obtain  the  same  result  as  we  obtain 
in  Corollary  7  below.  They  consider  the  case  of  deterministic 
workloads.  Gelenbe  [6]  introduced  an  alternate  model  for  the 
workload  for  which  he  also  calculates  speedup  in  the  case 
of  an  infinite  number  of  available  processors.  He  models  a 
job  as  having  a  random  task  graph  in  which  the  density  of 
precedence  relations  between  tasks  is  given  by  p  (0  <  p  <  I ); 
he  then  derives  an  approximation  for  an  uf^r  bound  on  the 
speedup;  namely,  (1  -F  p)/2p. 

II.  DEFiNmONS 

We  have  already  defined  the  following; 

P  =  Number  of  (identical)  processors  in  the  server. 

W  =  Average  number  of  seconds  required  to  process  a  job 
on  a  single  processor,  and 

y  =  Average  number  of  jobs  in  the  system. 
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Moreover,  we  now  define  the  following  additional  quanti¬ 
ties: 

x{  P)  =  Mean  service  time  of  a  job  in  a  P-processor 
system  (note  that  the  maximum  mean  service 
time  is  z(l)  =  W  and  that  the  minimum  mean 
service  time  is  x(oo)); 

T(X,  P)  =  Mean  response  time  (queueing  time  plus  service 
time)  of  a  job  in  a  queueing  system  with  an 
input  rate  A  and  P  processors; 

A  =  arrival  rate  of  jobs; 

p  =  system  utilization;  i.e.,  the  fraction  of  time 
when  there  is  at  least  one  job  in  the  system. 

=  Az(P);  and 

u(P)  =  processor  efficiency  in  a  P-processor  system. 

Note  the  difference  between  u(P),  which  is  the  average 
processor  efficiency  given  P  processors,  and  p,  which  is 
the  average  system  utilization.  Whenever  there  is  a  job  in 
the  system,  the  system  utilization  is  “1,”  but  the  processor 
efficiency  need  not  be  “1”  in  that  case,  since  there  may  be 
some  idle  processors  (i.e.,  it  may  be  that  the  job  in  service  does 
not  require  all  the  processors).  Hence  the  system  utilization  is 
always  greater  than  or  equal  to  the  processor  efficiency.  (Note 
that  u(l)  =  p  for  a  single  processor  queueing  system.) 

Two  cases  regarding  the  number  of  jobs  in  the  system 
are  considered  in  this  paper.  Case  one  allows  no  arrivals  of 
additional  jobs  (Section  IV).  That  is,  there  is  only  one  job  in 
the  system,  and  we  are  concerned  with  y(P),  its  mean  service 
time  in  a  P-processor  system.  Case  two  allows  jobs  to  arrive 
from  a  Poisson  process  at  a  rate  A,  and  so  queueing  effects 
are  considered  (Seaion  V). 

For  the  first  case,  we  define  the  (no  arrivals  case)  speedup 
with  P  processors,  denoted  by  S„{P),  to  be 

’  x(P)  x(P)- 

Note  that 

W  W 

1=  4tt  <5„(P)<  =P—. 

x(l)  x(oo) 

Thus  it  is  natural  for  us  to  define  the  maximum  value  for 
speedup  5„,max  as  follows: 

W 

— 

‘^n.max  —  — • 

x(oo) 

Furthermore,  we  see  that  5n  max  =  average  parallelism. 

For  the  second  case,  we  define  the  (arrivals  case)  speedup 
with  P  processors  at  system  utilization  p,  denoted  by  5a(  A,  P), 
to  be 


5,(A,P)  = 


7^(A,1) 
r(A.p)  ■ 


We  must  distinguish  the  processor  efficiency  u(P)  in  these 
two  cases  as  follows: 

Un(P)  =  processor  efficiency  given  P  processors  in  the 
no  arrivals  case;  and 

u„{\.P)  =  processor  efficiency  given  job  arrival  rate  A 
and  P  processors  in  the  case  with  job  arrivals. 


We  now  introduce  the  appropriate  definitions  of  power, 
which  we  denote  by  the  symbol  Q  (we  would  prefer  to  use 
the  obvious  notation  P,  but  P  has  already  been  used  to  denote 
the  number  of  processors).  Let 

Qn(P)  =  power  given  P  processors  in  the  no  arrivals 
case;  and 

Qo(A,  P)  =  power  given  a  job  arrival  rate  A  and  P  pro¬ 
cessors  in  the  case  with  job  arrivals. 

In  this  paper  we  are  concerned  mostly  with  power  which  is 
defined  as  processor  efficiency  divided  by  the  mean  response 
time. 

In  the  case  of  no  arrivals,  the  mean  response  time  of  the 
(single)  job  is  simply  its  mean  service  time  x{P),  and  so; 

Q  (P)=  ^n(P) 

Clearly,  power  will  increase  by  either  raising  the  processor 
efficiency  or  by  lowering  the  mean  service  time.  A  more 
general  definition  of  power  (as  originally  introduced  in  [12]) 
is  given  as 

^  ’  x(P) 

where  r  is  a  positive  real  number  whose  value  may  be  selected 
by  the  system  designer.  With  this  generalization,  a  designer 
may  express  a  stronger  preference  for  an  increase  in  the 
processor  efficiency  at  the  expense  of  an  increase  in  the  mean 
service  time  by  simply  increasing  the  value  of  the  parameter 
r  (and  vice-versa).  Note  that  Qn{P)  =  Qn  *(P)- 

In  the  case  of  job  arrivals,  the  definition  of  power  becomes; 

Qa(X,P)~ 

and  the  generalization  in  this  case  is 

Wa  (  .  )  r(A,P) 

where  again  r  is  a  positive  real  number  to  be  used  as  a  degree 
of  freedom  by  the  system  designer.  Note  that  Qa(A,P)  = 
Q1‘^(A,P). 

With  these  definitions  of  power,  our  goal  is  to  find  the 
optimal  number  of  processors  to  use  in  a  parallel  processing 
system  such  that  power  is  maximized.  Furthermore,  in  the  case 
of  job  arrivals,  we  also  seek  the  optimal  system  operating  point 
(i.e.,  the  optimal  input  rate  of  jobs). 

The  rest  of  this  paper  is  organized  as  follows.  In  Section  111 
we  present  two  models  of  a  job:  a  discrete  model,  and  a 
continuous  model.  In  Section  IV  we  solve  the  case  when 
no  arrivals  are  allowed  in  the  system.  In  this  case  we  find 
the  speedup  of  the  system  given  P  processors.  We  also  find 
P",  the  number  of  processors  which  maximizes  power.  In 
Section  V  we  solve  the  case  when  job  arrivals  are  allowed 
in  the  system.  In  this  case  we  again  solve  for  the  speedup 
of  the  system  given  P  processors.  We  also  find  A’  and  P*, 
which  maximize  power.  One  interesting  result  we  get  is  that 
the  P*  for  systems  with  no  arrivals  and  the  P*  for  systems 
with  arrivals  are  equal  when  power  is  maximized;  this  provides 
a  simplification  in  system  design. 
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III.  Workload  Models 

We  consider  both  a  discrete  as  well  as  a  continuous  model 
of  job  requirements. 

A.  A  Discrete  Job  Model 

Here,  we  model  a  job  as  containing  a  total  of  U'  tasks. 
Nonoverlapping  subsets  of  these  tasks  are  collected  into  stages, 
and  these  stages  are  processed  sequentially  ( however^aral- 
lelism  is  exploited  within  each  stage — see  below).  W  is  a 
random  variable  with  mean  IV'  and  coefficient  of  variation 
civ We  assume  that  the  service  time  distribution  for  each 
task  is  deterministic,  such  that  each  task  requires  1  s  of  work 
on  a  processor.  For  the  results  we  seek  in  this  paper,  a  job 
is  described  by  specifying  VV^  and  cw  along  with  two  other 
vectors.  The  first  vector  is  called  the  fraction  vector,  /',  and 
the  second  vector  is  called  the  processor  vector,  P'.  We  denote 
the  fraction  and  processor  vectors  as 

/'  =  [/(. /^ /a, 

where  n'  is  the  number  of  stages  in  a  job.  The  ith  stage  has 
the  pair  (/,^  P/)  associated  with  it.  The  meaning  is  as  follows: 
a  fraction  f-  of  the  total  tasks  in  a  job  can  use  P/  processors 
to  concurrently  process  these  tasks.  For  this  definition,  it  is 
clear  that 

n' 

i=l 

The  example  from  Fig.  1  is  repeated  in  Fig.  2(a),  where 
W  =  24  and  n'  =  6.  Stage  4  contains  12  tasks,  and  so 
=  1/2;  moreover,  since  PJ  =  6  (and  if  P  >  6),  then  it 
will  take  2  s  to  complete  stage  4.  This  stage-type  workload 
model  comes  directly  from  the  usual  task  graph  model  of 
computation  [3]  with  deterministic  task  service  times.  The  tth 
stage  conesponds  to  the  ith  level  in  the  computation  graph. 

For  convenience,  we  may  rearrange  the  elements  in  /'  and 
P'  as  follows  in  such  a  way  that  neither  the  mean  response 
time  nor  the  processor  efficiency  are  changed.  The  elements 
of  P'  are  rearranged  and  renumbered  so  that  its  elements  are 
nondecreasing;  that  is,  P/_i  <  P-.  The  elements  of  f  follow 
the  identical  permutation  and  renumbering.  We  may  then 
merge  several  stages  with  the  same  P-’s  into  one  stage  simply 
by  adding  all  the  corresponding  //’s.  The  new  vectors  will 
be  denoted  P  =  [Pi,  P2,  •  •  • ,  Pn]  and  /  =  [A, /2,  • ,  fn], 

where  n  <  n'  and  Pi_i  <  P,-.  Since  the  system  admits 
only  one  job  into  service  at  a  time,  it  can  easily  be  shown 
that  this  rearrangement  does  not  affect  the  performance  at  ail. 
The  example  in  Fig.  2(a)  has  been  rearranged  as  shown  in 
Fig.  2(b),  where  the  number  of  stages  is  now  n  =  4.  Note 
that  z(oo)  =  8,  as  it  was  in  Fig.  2(a).  One  can  easily  see  that 
if  we  choose  P  =  4,  then  x(4)  will  equal  9  in  this  rearranged 
case,  as  was  the  case  for  Fig.  1(b). 

'The  coefRcietit  of  viriaikM  of  a  random  variable  is  equal  to  its  standard 
deviation  divided  by  its  mean. 


Fig.  2.  Rearranging  the  job  profile,  (a)  P  ss  [2,4, 2,6,2,  !),/=:[ 
^,y,^,^].(b)P  =  [l,2,4,6],  /  = 

B.  A  Continuous  Job  Model 

We  now  describe  a  continuous  version  of  the  above  model. 
In  this  model  we  assume  that  the  number  of  processors 
required  by  jobs  is  a  (not  necessarily  discrete)  nondecreas¬ 
ing  function  of  time  (recall  the  rearranging  does  not  affect 
performance).  That  is,  we  permit  nonintegral  numbers  of 
processors  (which  could  correspond  to  cases  where  processors 
are  shared  among  more  than  one  job).  A  special  model  with 
a  deterministic  workload  per  job  will  be  described  first,  and 
then  a  more  general  model  with  a  random  workload  per  job 
will  be  described. 

For  the  special  case  with  a  deterministic  workload,  we 
define  P(t)  =  g(t),  where  g{t)  is  a  deterministic  function, 
to  be  the  number  of  processors  that  a  job  desires  at  time 
t  (0  <t  <b)  such  that  P(6)  =  B  (see  Fig.  3).  For  such  a 
model,  the  workload  (seconds  of  work  required)  for  each  job 
is  deterministic  with  value 

i 

W  =  j  P{t)dt. 

0 

Note  that  b  =  x(<x>).  Moreover,  if  we  limit  the  number  of 
processors  to  P(P  <  B),  then  A,  the  (shaded)  area  of  Pin 
which  lies  above  the  value  of  P,  will  be  flattened  out  and 
extended  as  a  rectangle  of  area  A  and  of  height  P  beginning 
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Abstract 

We  use  simple  models  of  two  distributed 
processing  methods,  one  asynchronous,  the 
other  synchronous,  to  calculate  the  maxi¬ 
mum  potential  performance  gain  of  the  for¬ 
mer  over  the  latter.  We  show,  in  the  limit 
as  the  number  of  tasks  grows  and  the  num¬ 
ber  of  processors  increases,  that  the  asyn¬ 
chronous  method  has  an  expected  potential 
speedup  over  the  synchronous  method  of  no 
more  than  In  P  where  P  is  the  number  of 
processors  used  by  each  strategy. 

1  Introduction 

We  compare  two  synchronization  methods  used  in 
distributed  processing  systems  and  determine  how 
much  better  one  performs  than  the  other.  Our  moti¬ 
vation  comes  from  the  area  of  Parallel  Discrete  Event 
Simulation  (PDES)  which  has  received  much  atten¬ 
tion  recently  [Misra  1986]  [Jefferson  1985].  There 
are  several  algorithms  used  for  PDES  and  this  paper 
demonstrates  the  potential  improvement  by  using  an 
asynchronous  approach  (e.g.  Time  Warp),  over  a 
synchronous  technique,  (e.g.  time-stepped  simula¬ 
tion).  We  first  give  an  introduction  to  PDES,  discuss 
briefly  the  two  methods  chosen  for  compariscm,  and 
then  follow  with  our  models  and  analysis.  Readers 
who  are  unfamiliar  with  Discrete  Event  Simulation 
and  techniques  used  to  parallelize  it  are  referred  to 
[Misra  1986]  [Jefferson  1985]  [Peacock  et  al.  1979] 
for  more  details. 

2  Parallel  Discrete  Event 
Simulation 

Parallel  Discrete  Event  Simulati<Mi  is  generally  ac¬ 
complished  by  partitioning  the  simulation  into  lop- 
cal  processes  (LP)  or  entiiies  each  of  which  simulates 
some  physical  process  in  the  system.  An  example 

"This  work  was  supported  by  the  Defense  Advanced 
Research  Projects  Agency  under  Contract  MDA  903-87- 
C0663,  Parallel  Sjrstems  Laboratory. 
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Figure  1:  Example  queueing  network 


is  the  simple  queueing  network  shown  in  Figure  1. 
Entities  in  our  system  are  the  customer  arrival  pro- 
ces8(A),  the  servers(B,C,D,E)  and  a  final  sink  pro- 
ces8(F)  to  collect  departing  customers.  Each  process 
receives  messages,  performs  internal  computations 
and  sends  messages  to  other  processes.  Each  LP 
maintains  a  local  clock  which  indicates  the  current 
time  of  the  simulation  at  that  entity,  and  a  process 
terminates  once  its  local  or  logical  clock  (the  simula¬ 
tion  time  of  the  message  currently  being  processed) 
has  reached  Tmar,  the  total  time  of  the  simulation  (a 
user  specified  duration).  One  can  think  of  each  log¬ 
ical  process  as  residing  on  a  separate  processor,  but 
this  is  not  necessary.  In  fact,  all  the  logical  processes 
may  reside  on  a  single  processor.  LPs  operate  in¬ 
dependently  and  communicate  with  each  other  only 
if  the  physical  processes  being  simulated  by  the  LPs 
ate  connected.  For  example,  logical  process  A  (LP^) 
connects  to  LPa  which  is  in  turn  connected  to  LPc 
and  LP|>  etc.  Every  path  which  can  be  traversed  by 
a  customer  in  the  physical  system  must  c(»respond 
to  a  logical  communication  path  in  the  simulation 
system.  Messages  passed  between  LPs  in  our  queue¬ 
ing  example  are  the  actual  customers  flowing  through 
the  system. 

Each  logical  process  could  be  placed  on  its  own 
processor,  and  one  might  hope  that  we  could  then 
gain  speedup  prop<»tional  to  the  number  of  proces¬ 
sors  used.  Unfortunately,  this  is  often  not  the  case 
as  the  system  being  simulated  may  have  only  limited 
parallelism.  Also,  the  PDES  algorithms  themselves 
limit  parallelism  in  their  attempt  to  prevent  the  sim¬ 
ulation  from  deadlocking  and  to  ensure  correctness. 
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Several  competing  techniques  have  been  developed 
to  address  deadlocking  and  correctness.  One  [Pear 
cock  et  al.  1979]  is  described  as  a  synchronous  ap¬ 
proach  which  keeps  logical  process  clocks  in  synchro¬ 
nization  while  another  [Jefferson  1985]  is  an  asyn¬ 
chronous  strategy  which  uses  a  rollback  mechanism 
which  is  invoked  only  when  needed  for  synchroniza¬ 
tion. 

2.1  Time-Stepped  Simulation 

Distributed  time-stepped  simulation  [Peacock  et  al. 
1979]  is  accomplished  by  keeping  all  the  local  clocks 
in  strict  synchronization.  At  any  point  in  real  time 
each  LP’s  local  clock  will  have  the  same  value  as 
any  other  LP’s  clock.  As  the  simulation  runs,  the 
loc2d  clocks  take  on  a  sequence  of  discrete  values 
(to.fiifz. •  •  •)  each  differing  by  an  amount  A.  All 
processors  must  complete  execution  of  events  up  to  U 
before  any  processor  begins  processing  at  U+i.  Each 
processor  may  have  a  different  amount  of  work  to 
do  at  each  time  step  or  some  may  operate  at  differ¬ 
ent  speeds  so  that  many  processors  may  have  to  wait 
for  the  slowest  one  to  complete  execution  of  the 
step.  Time-stepped  simulation  is  attractive  due  to 
its  simplicity  of  implementation.  By  keeping  all  the 
LPs  processing  at  the  same  simulation  time,  dead¬ 
locks  cannot  occur  and  no  further  effort  needs  to  be 
expended  in  guaranteeing  the  correctness  of  the  sim¬ 
ulation.  Time-stepped  simulation  is  an  example  of 
the  synchronous  approach. 

2.2  Time  Warp 

Our  asynchronous  example  comes  from  one  of  the 
more  recent  developments  in  the  area  of  PDES;  the 
so-called  optimistic  strategies.  One  such  strategy  is 
called  Time  Warp  and  was  developed  by  Jefferson 
[Jefferson  1985].  The  basic  idea  is  that  the  require¬ 
ment  of  keeping  each  LP  in  strict  synchronization 
(keeping  local  clocks  the  same),  even  when  it  isn’t 
necessary,  may  lead  to  a  degradation  in  performance. 
The  Time  Warp  mechanism  allows  LPs  to  race  for¬ 
ward  as  quickly  as  possible.  If  a  message  arrives 
which  has  a  lower  timestamp  than  the  value  of  the 
LP’s  clock,  indicating  the  LP  proceeded  with  incom¬ 
plete  information,  tlw  LP  is  “rolled  back”  to  the  time 
of  the  incoming  message.  This  can  be  accomplished 
because  the  system  periodically  saves  the  state  of  the 
LP.  Any  effects  of  having  advanced  too  far  (i.e.  er¬ 
roneous  messages)  are  canceled  through  an  elegant 
technique  using  anti-messages  [Jefferson  1985]. 

3  The  Models 

We  have  opted  to  use  very  simple  models  of  the  two 
approaches  in  order  to  assess  the  potential  improve- 
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Figure  2:  Synchronous  Task  Graph 


ment  of  the  asynchronous  versus  the  synchronous 
strategy.  Our  model  of  the  time-stepped  (syn¬ 
chronous)  strategy  will  provide  us  with  an  accurate 
estimate  of  its  time  to  complete  a  simulation,  while 
the  model  for  the  asynchronous  strategy  will  provide 
us  with  an  overly  low  estimate  of  its  expected  comple¬ 
tion  time.  Therefore,  we  establish  an  upper  bound 
on  the  potentiad  improvement  of  the  asynchronous 
strategy  over  the  time-stepped  method. 

To  analyze  the  two  techniques,  we  propose  the  fol¬ 
lowing  model:  P  processors  eadi  execute  K  tasks 
(events)  sequentially.  Each  processor  p  must  per¬ 
form  tasks  i],i  •  •  •  Tpt  •  •  •  TpK  in  sequential  order.  K 
determines  the  “size”  of  the  simulation.  A  task  will 
take  a  random  amount  of  time  to  complete  execution 
on  any  processor. 

Our  model  of  the  synchronous  approach  is  based 
on  the  idea  that  an  LP  must  wait  for  all  other  LPs 
to  complete  a  step  before  continuing.  Each  processor 
must  wait  until  every  processor  has  completed  its 
P*  task  prior  to  beginning  execution  of  the  (t  -I- 1)** 
task.  This  is  essentially  a  “staged”  execution  with  K 
stages  where  each  stage  takes  as  long  as  the  slowest 
processor.  This  task  graph  is  shown  in  Figure  2  for 
X  =  4  and  P  =  7. 

The  asynchronous  strategy  has  no  such  ”staging” 
restriction,  and,  moreover,  in  the  best  possible  cir¬ 
cumstance  no  rollbacks  will  occur.  We  allow  each 
processor  to  execute  its  tasks  in  order  as  fast  as  it 
can,  without  waiting  for  the  other  processors  to  fin¬ 
ish.  The  total  time  to  finish  is  simply  the  time  that 
the  slowest  processor  takes  to  complete  its  K  tasks. 
To  keep  the  model  simple  we  are  assuming  no  roll¬ 
backs;  it  is  as  if  each  processor  never  has  to  wait  for 
any  messages  from  other  processors,  and  that  all  mes¬ 
sages  arrive  in  timestamp  order.  The  asynchronous 
task  graph  is  shown  in  Figure  3. 


132 


^ - K-4 - - 

D  -  No  execution  time,  synchronization  only 
O  *  A  task 


Figure  3:  Asynchronous  Task  Graph 

4  Space  of  Synchronization 
Methods 


Using  the  PDF  we  can  calculate  the  expected  value 
of  T  [Kleinrock  1975]. 

EIT\  =  /“(I  -  FT{x))dx 

Jo 

=  e(  ^ 

Since  [Graham  et  al.  1989] 


Though  the  models  we  are  using  are  extremely  sim¬ 
ple,  we  believe  they  provide  us  with  very  impor¬ 
tant  information.  Our  time-stepped  model  (Fig¬ 
ure  2)  requires  the  most  synchronization,  and  there¬ 
fore  will  take  the  longest  time  to  complete  execution 
of  any  system  which  exhibits  full  parallelism  in  each 
stage.  Our  asynchronous  model  (Figure  3)  shows  the 
least  amount  of  internal  synchronization  (none)  and 
should  complete  execution  in  less  time  than  any  other 
method.  Therefore,  we  believe  that  these  two  models 
span  the  range  of  possibilities  and  give  a  good  indica¬ 
tion  of  the  maximum  performance  improvement  that 
could  be  gained  by  using  the  asynchronous  strategy. 

5  Exponentially  Distributed 
Task  Times 

If  we  model  each  task  execution  time  with  an  expo¬ 
nential  distribution  and  treat  the  processors  as  iden¬ 
tical,  the  expected  time  for  the  synchronous  strategy 
is  K  times  t^  maximum  of  P  exponentials,  while  the 
expected  time  for  the  asynchronous  strategy  is  the 
maximum  of  P  Hf-stage  Erlangs.  We  now  proceed 
to  calculate  the  ratio  of  the  expected  conqiletion 
times  for  exponentially  distributed  task  times. 

5.1  Time-9Cipq;>ed  (Synchronous) 
Model  ' 

Let  T  —  the  maximum  of  P  exponentials  with  mean 
1.  The  cumulative  distribution  (PDF)  of  T  is 

/ir(*)  =  (l~e-n^  (1) 

with  density  function 

/,(*)  = /’(I  - (2) 


We  now  define  T,  as  the  time  for  all  K  stages  to 
complete.  Clearly,  £[T«]  =  KE[r\.  So  the  final 
equation  for  E[Tt]  where  P  is  the  number  of  pro¬ 
cessors  and  K  is  the  number  of  steps  is: 

^^P'-1  =  7E7-  <') 

An  excellent  iq>proximation  for  this  is  [Jolley  1961]: 

+  (5) 

where  E  =  Euler’s  Constant  »  0.57722. 


5.2  Time  Warp  (Asynchronous) 
Model 


We  define  Ta  as  the  maximum  of  P  A'-stage  Erlangs 
where  each  stage  has  mean  The  probability  den¬ 
sity  function  of  a  single  A'-stage  Erlang  is 


/.(*) 


pe”'**(/i*)*~‘ 

(K-iy. 


(6) 


The  PDF  can  be  found  either  by  direct  integration  of 
the  density  function,  or  by  realizing  that  the  prob¬ 
ability  that  a  A’-stage  Erlang  takes  time  less  than 
or  equal  to  f  is  one  minus  the  probability  that  it 
takes  time  greater  than  i,  which  is  simply  one  minus 
the  probability  that  there  are  leas  than  K  arrivals  in 
the  interval  [0  —  f]  from  a  Poisson  process  at  rate  /i. 
Therefore 
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Te  Versus  P 


P  (Number  of  Processors) 
(log  scale) 


Regression  Slope  values  versus  Ln(K)‘2 


4  6 

Ln(K)*2 


Regression  Intercept  Values  versus  K 


Figure  4:  T*  versus  In  P. 

The  cumulative  distribution  of  the  maximum  of  P 
A'-stage  Erlangs  is; 

Ft.(*)=  .  (8) 

Using  we  can  calculate  the  expectation  of  T,. 

E[T,]=  r[l-FTA^)]dx 

Jo 

Thus, 

Unfortunately,  this  equation  has  no  closed  form  ex¬ 
pression  for  the  integral.  By  decomposing  £[7'a]  into 
two  components:  the  mean  of  s  ff-stage  Erlang  and 

Te  =  the  diiference  between  the  mean  and  the  ex¬ 
pected  value  ci  the  max  of  P  Af-stage  Erlangs,  we 
cam  approximate 

E[Ta]  =  Mean  of  AT-stage  Erlang  +  Te 

=  — +  T. 

P 

An  excellent  approximation  for  Te  when  Af  >  1  and 
P  >  1  is 

r,  «  i((CTn’A'-»-Z?)lnP  +  AAf -l-B).  (9) 


5  10  IS 

K  (Number  of  Steps) 


Figure  5:  Regression  Values 


Where 

A  =  0.02244  ss  0.02  B  =  1.14571  «  1.15 
C  =  0.22278  «  0.22  D  =  0.55957  «  0.56. 

This  approximation  was  developed  by  using  least 
squares  regression  techniques  three  times.  It  was  first 
noticed  that  for  a  fixed  K,  Tg  seemed  to  be  directly 
related  to  In  P.  This  is  clearly  seen  in  Figure  4.  For 
each  value  of  A  <  Af  we  performed  a  linear  regres¬ 
sion  for  Te  such  that  {Te)^  =  mt(lnP)  +  6^,  thus 
generating  K  slopes  and  intercepts,  one  set  for  each 
value  of  k.  Then,  it  appeared  that  the  slopes  for  each 
k  approximation  were  linearly  related  to  In^b,  while 
the  intercepts  seemed  linearly  related  to  k.  This  can 
be  seen  in  Figure  5.  Therefore,  a  second  recession 
was  performed  on  the  slope  values  versus  In^fc  (gen¬ 
erates  the  values  for  the  constants  C  and  D),  while  a 
third  regression  was  performed  on  the  intercept  val¬ 
ues  versus  k  (generates  the  values  for  the  constants 
A  and  B).  Figure  6  shows  the  approximation  com¬ 
pared  to  simulation  for  values  of  K  and  P  between 
one  and  ten  and  Figure  7  shows  the  comparison  for 
K,P>=  100. 

5.3  Relative  Performance 

Let  us  look  at  the  ratio,  Rg  of  the  two  approxima¬ 
tions. 


£[r,i»A( 


E  +  \aP+^p 
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Comparison  of  E[Ta]  Approximation  with 
Simulation  (Confidence  98X)  (p  -  1/4) 


0  2  4  6  a  10 

P  (Number  of  Processors) 


Figure  6;  Comparison  of  Approximation  and  Simu¬ 
lation  for  K  <  10. 


Figure  7:  C!<»nparison  of  Approximation  and  Simu¬ 
lation  for  K  >  1(X). 


F(T«]  «  ^  ^  {(Cla^K  +  D)lnP  +  AK-^-B) 

R  - 

^  "  E[Ta] 

^  (g  -f  ln(P)  4-  ^ 

i  ((CTln^C AT)  -I-  D)  ln{P)  -I-  ( 1  -I-  A) /C  -|-  S) 


£  -1-  In 

=  JcJE!E+MIT(i7A)TJ 

Taking  the  limit  as  the  size  of  the  simulation  in¬ 
creases  (K  —>■  oo)  and  assuming  that  K  »  InP 
we  get 

E  +  inP+^~  pII^h 

- (ITm) - 

Finally,  for  large  P 

lun  /2e  «  .  ..  «  TT^r  «  In  P.  (10) 

K-oo.P-oo  ^  (1  +  A)  1.02 


Thus,  in  the  limit  as  the  size  of  the  simulation  in¬ 
creases  to  infinity,  the  asynchronous  approach  could, 
at  most,  complete  InP  times  as  fast  as  the  time- 
stepped  method  on  average.  We  can  derive  this  re¬ 
sult  by  appealing  to  intuition.  We  have  exponential 
task  times  where  each  task  takes,  on  average,  ^  sec¬ 
onds  to  complete.  For  synchronized  execution,  basic 
principles  (^tion«5.1)  tell  us  that  each  stage  will 
take  time  proportional  to  In  P  on  the  average  for 

a  total  expected  time  of  ^  In  P.  The  asynchronous 
execution  on  average  takes  time  equal  to  ^  plus  Te 
a  term  which  is  small  compared  to  ^  for  large  K. 
Therefore,  the  ratio  of  the  two  times  should  be  In  P. 
It  should  be  noted  that  a  trivial  lower  bound  on  Ta 
(thus  an  upper  bound  on  R,)  is  found  by  simply  using 
the  mean  of  P  ff-stage  Erlangs.  Our  approximation 
(Equation  9)  confirms  this  result  since  A,B,C  and  D 
are  all  non-negative. 

Additionally,  if  we  believe  that  no  method  could 
achieve  a  spe^up  greater  than  P  for  P  processors 
over  executxm  on  a  single  processor,  then  any  time- 
stepped  method  is  limited  to  a  maximum  speedup 
of  These  results  depend  on  the  assumptkMi  of 
an  exponential  distribution  for  task  times,  'ne  next 
section  uses  a  uniform  [0-X]  distribution  for  task  ex¬ 
ecution  times. 


6  Uniformly  Distributed  Task 
Times 

If  we  make  the  assumption  that  the  task  times  are 
uniformly  distributed  between  0  and  X,  we  calculate 
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a  different  limiting  value  for  the  ratio  of  completion 
times.  It  is  easy  to  show  that  the  maximum  of  P 
uniformly  distributed  random  variables  is  X  .  We 
immediately  find  that 

E[T.]  =  KX-^.  (11) 

Fortuitously,  we  can  use  the  seime  regression  tech¬ 
nique  used  with  the  exponential  distribution  in  Sec¬ 
tion  5.2  to  develop  an  accurate  approximation  for 
^[Ta].  Therefore 


average  for  a  total  time  of  KX.  The  asynchronous 
system,  on  average,  will  take  time  equal  to  -t-Tj. 
Since  T*  is  small  compared  to  for  large  K,  the 
speedup  ratio  should  be  2. 

As  before,  if  we  assume  that  no  method  can  achieve 
speedup  greater  than  P  over  a  sequential  execution, 
then  the  synchronous  strategy  could  possibly  have 
speedup  proportional  to  P  when  the  task  times  are 
uniformly  distributed. 

7  Conclusions 


KX 

w  —  +  X  +  D)\n  P  +  AK  B) .  (12) 

Where 


A  =  0.012384  a  0.01  B  =  0.330691  «  0.33 
C  =  0.053147  «  0.05  D  =  0.125102  a  0.13. 


Finally,  we  look  at  the  ratio  of  the  expected  com¬ 
pletion  times. 


_  E[T,] 
E[Ta] 


XKj^ 

X  ((Cln*(A’)  +  D)  \n{P)  +  {A+  l/2)K  -1-  B) 


p 

_ _ HI _ 


Taking  the  limit  as  the  size  of  the  simulation  in¬ 
creases  {K  — ►  oo)  and  assuming  that  K  »  InP 
we  get 

„  P 

««  -  (>i  +  i/2)(P-H)- 

Finally,  for  large  P 


Urn  = 

K—oo.P—oo 


(A  +  1/2)  0.51 


(13) 


We  have  shown  that  an  asynchronous  distributed 
simulation  strategy  can  have  at  most  a  InP  per¬ 
formance  improvement  over  a  time-stepped  method, 
in  the  case  where  task  times  are  exponentials.  We 
conjecture  that  this  result  is  due  to  the  infinite  tail 
on  the  exponential  distribution  and  may  therefore 
be  applicable  to  distributions  with  exponential  tails. 
The  improvement  when  using  a  distribution  with  fi¬ 
nite  support  (e.g.  uniform)  is  reduced  to  a  constant 
aunount  independent  of  P. 
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Thus,  when  using  a  uniform  distribution,  the  asyn¬ 
chronous  strategy  is  only  able  to  complete  in  rou^ly 
half  the  time,  regardless  of  the  number  of  processors 
used  (as  compared  to  our  previous  case  where  the 
task  execution  times  had  exponential  tails  and  the 
asynchronous  strategy  was  able  to  gain  its  In  P  per¬ 
formance  improvement).  This  result  should  apply  to 
any  distribution  with  finite  support  since  the  maxi¬ 
mum  of  many  such  random  variables  will  invariably 
approach  the  upper  limit. 

Again,  we  can  appeal  to  intuition  to  find  the 
speedup  ratio.  For  large  P  the  synchronized  exe¬ 
cution  wiU  take  X  seconds  per  stage  (the  max)  on 
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1.  Introduction 


The  design  and  performance  evaluation  of  dis¬ 
tributed  systems  is  an  important  and  difficult 
problem  and  one  which  will  occupy  our  attention 
for  the  next  decade.  Indeed  it  represents  an  exam¬ 
ple  of  the  class  of  resource  allocation  problems 
with  which  we  have  been  wrestling  for  many  years 
in  a  variety  of  different  contexts.  For  example,  the 
problem  of  designing  the  operating  system  for 
time-shared  computers  was  a  major  issue  in  the 
1960’s,  the  irsue  of  wide-area  network  design  and 
access  occupied  our  interest  in  the  1970’s,  the 
problem  of  local  area  network  design  was  our 
focus  for  the  decade  of  the  1980’s  and  we  foresee 
that  the  general  problem  of  distributed  processing 
will  surely  occupy  our  attention  for  the  coming 
decade  of  the  1990’s. 

One  aspect  of  the  problem  has  to  do  with 
resolving  conflicts.  This  issue  manifests  itself  both 
in  centralized,  as  well  as  in  distributed  systems. 
The  problem  arises  when  more  than  one  user 
requires  access  to  the  same  resource  at  the  same 
time.  Usually  we  cannot  predict  exactly  when  a 
user  will  require  access  to  the  resource,  we  cannot 
predict  how  long  each  user  will  hold  the  resource 
once  he  gains  access,  most  users  only  require  the 
occasional  use  of  the  resource,  and,  in  addition, 
when  a  user  asks  for  access  he  usually  expects 
immediate  access  to  that  resource.  This  presents  a 
nasty  set  of  requirements  on  the  part  of  the  user 
and  we  refer  to  such  a  class  of  users  as  being 
bursty  and  asynchronous.  There  are  four  canonical 
ways  of  resolving  conflicts.  The  first  is  queueing: 
here  one  user  gets  access  to  the  resomce  while  the 
others  wait  for  their  turn.  The  second  resolution 
method  is  that  of  splitting:  here,  the  resource  is 
split  into  as  many  pieces  as  there  are  competing 
users  and  each  user  gets  a  piece  of  the  resource. 
The  third  canonical  resolution  method  is  blocking: 
here,  one  user  gets  access  to  the  resource  and  the 
others  are  asked  to  go  away.  The  fourth  method  is 
smashing:  here,  if  more  than  one  user  asks  for 
access,  no  one  is  given  access.  Examples  of  each  of 
these  systems  are  prevalent  throughout  the  com¬ 
puter  and  communication  industry.  Of  course,  one 
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may  use  hybrid  mixtures  of  these  four  canonical 
resolution  methods. 

Queueing  is  perhaps  the  most  common  conflict 
resolution  method  and  is  often  found  in  our  cur¬ 
rent  technology.  We  are  all  familiar  with  the  price 
one  pays  for  queueing,  namely,  an  increase  in 
response  time  due  to  the  sharing  of  a  resource 
with  other  users  (see  for  example,  [1,2]).  However, 
in  a  distributed  system,  we  are  confronted  with 
additional  access  problems  and  delays  beyond 
those  due  to  pure  queueing.  This  comes  about  for 
many  reasons  discussed  in  the  next  section,  all 
related  to  the  fact  that  one  cannot  form  a  queue 
for  free  in  a  distributed  environment.  In  particu¬ 
lar,  we  will  focus  later  in  this  paper  on  the  specific 
issue  of  synchronization  among  coupled  processes. 

2.  Problems  of  Distributed  Access 

Once  we  distribute  resources  and  users,  we  are 
faced  with  a  number  of  difficult  conflict  resolution 
and  access  problems.  One  way  to  divide  these 
problems  into  recognizable  systems  is  to  consider 
a  two-dimensional  description  where  the  first  di¬ 
mension  describes  the  degree  of  coupling  among 
the  distributed  processes  (i.e.  the  amount  of  com¬ 
munication  and  interaction  among  them)  and 
where  the  second  dimension  describes  the  distance 
that  separates  these  processes.  This  may  be  seen  in 
Fig.  1. 

In  this  figure,  the  items  above  the  dashed  line 
within  a  quadrant  refer  to  processing  applications, 
whereas  items  below  the  dashed  line  refer  to  com¬ 
munications  applications.  Applications  in  the  case 
of  tightly  coupled  processes  which  are  close  to 
each  other  include  parallel  processing  (a  number 
of  processors  cooperating  in  the  execution  of  a 


COl 

PUNG 

TIGHT 

LOOSE 

DISTANCE 

CLOSE 

PARALLEL  PROCESSING 

DISTRIIUTEO  PROCESSING 

SACK  PLANE  BUS 

HIGH  SPUD  LANs 

LANs 

MANS 

FAR 

HAI 

OISTRISUTEO  ACCESS 

WANs 

Fig.  1.  Examples  of  distributed  systems. 


single  problem)  as  well  as  back  plane  buses  and 
high  speed  LANs.  In  the  case  of  loosely  coupled 
processes  which  are  close  to  each  other,  we  include 
such  things  as  distributed  processing  (a  number  of 
processors,  each  possibly  working  on  a  different 
problem  with  occasional  interaction  among  these 
processors)  as  well  as  LANs  and  MANs.  Loosely 
coupled  processes  which  are  separated  by  large 
distances  include  applications  such  as  distributed 
access  (e.g.  remote  access  to  a  data  base)  as  well  as 
wide-area  networks.  However,  in  the  case  of  tightly 
coupled  processes  which  are  far  from  each  other, 
we  find  very  few  applications,  all  of  which  are 
extremely  difficult  due  to  the  large  amount  of 
interaction  required  at  what  may  be  long  distances 
and/or  long  delays. 

A  major  problem  that  we  face  in  distributed 
access  is  that  we  usually  lack  global  knowledge 
regarding  the  system  state.  A  number  of  things 
contribute  to  this  lack  of  global  knowledge  and 
manifest  themselves  as  problems.  For  example,  a 
long  distance  between  users  which  must  interact 
becomes  a  problem  with  regard  to  the  speed  with 
which  they  can  interact  and  the  bandwidth  of  the 
communications  of  that  interaction  (for  example, 
it  takes  light  approximately  150(X)  microseconds 
to  cross  the  United  States!);  thus,  the  time  for 
state  information  to  be  exchanged  between 
processes  may  be  seriously  delayed  leading  to  a 
lack  of  global  knowledge.  Another  problem  is  that 
not  all  processes  (users)  may  be  in  immediate 
communication  with  each  other;  for  example,  two 
users  may  not  be  able  to  hear  each  other  and  may 
require  intermediate  users  to  relay  information 
between  them.  Sometimes  the  state  information 
we  get  is  incomplete  and  sometimes  it  is  incorrect. 
Even  if  it  is  complete  and  correct,  it  may  be  that 
the  state  information  is  stale  by  the  time  we  have 
an  opportunity  to  use  it,  since  it  may  take  a  user  a 
certain  amount  of  time  to  get  to  the  location 
where  he  can  use  the  information  he  gathered 
previously.  Thus,  lack  of  global  knowledge  pre¬ 
vents  perfect  use  of  all  system  resources  in  ad¬ 
dition  to  any  congestion  problems  which  would 
arise  in  a  centralized  conEguration  with  perfect 
knowledge. 

Another  source  of  difficulty  in  distributed  sys¬ 
tems  has  to  do  with  access  to  resources  them¬ 
selves.  For  example,  even  though  there  is  a  set  of 
resources  in  a  system,  not  all  users  may  be  allowed 
access  to  certain  of  the  resources.  Livny  and  Mel- 
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Fig.  2.  Performance  of  S  M/M/1  queueing  systems. 


man  [3],  studied  the  case  of  N  independent 
M/M/1  queueing  systems,  each  with  its  own  in¬ 
dependent  arrival  process  at  rate  \  and  each  with 
its  own  server  at  rate  /i.  For  this  system,  the 
probability  Q  that  at  least  one  customer  is  waiting 
in  some  queue  and  at  least  one  server  is  idle  is 
given  by 

e=l-p"'  +  (l-p)^[p^-(l+p)^].  (2.1) 

In  Fig.  2,  we  plot  0  as  a  function  of  p  =  X/p 
with  Xf  as  a  parameter.  From  this  result  we  see 
that  Q  approaches  I  for  ail  values  of  p  in  the 
range  0  <  p  <  1,  as  the  number  of  independent 
systems  increases  (N  -*  oo)  indicating  serious  de¬ 
gradation  to  system  performance.  Of  course,  this 
is  only  a  simple  measure  of  system  degradation 
but  does  indicate  the  cost  of  prohibited  access. 

One  way  to  improve  performance  of  these  N 
independent  queueing  systems  is  to  allow  some 
jockeying  among  the  queues.  A  model  which  in¬ 
troduces  this  jockeying  is  to  allow  each  user  from 
queue  n  (n  =  1,  2, . . . ,  iV)  to  move  to  queue  «  -I- 1 
(mod  Xf)  at  a  rate  a.  Qearly,  for  a  =  0,  we  have 
Livny’s  model  and  for  a  =  oo  we  have  a  simple 
M/M/Xf  queue  which  provides,  in  some  sense, 
perfect  sharing  of  the  N  servers.  Unfortunately, 
this  model  with  jockeying  is  an  unsolved  coupled 
queueing  problem  (even  for  the  case  N  »  2);  cou¬ 
pled  queueing  problems  are  very  difflcult  in  gen¬ 
eral. 

Another  problem  with  access  has  to  do  with  the 
fact  that  the  distributed  system  may  be  lossy;  by 
this  we  mean  it  is  possible,  that,  as  the  load 
increases,  the  throughput  might  decrease  due  to 
internal  waste  of  resources  (see  for  example  [4]). 

The  access  problem  may  also  manifest  itself,  in 
some  cases,  by  requiring  that  resources  be  used  in 


series  (one  after  the  other)  rather  than  all  at  once. 
This  leads  to  increased  delays  and  possibly  ineffi¬ 
cient  use  of  resources.  An  example  here  might  be 
that  of  a  message  passing  through  a  wide-area 
network  which  must  hop  from  channel  to  channel 
as  it  makes  its  way  through  the  network. 

Another  manifestation  of  this  access  problem  is 
that  there  may  be  synchronization  and/or  prece¬ 
dence  constraints  in  the  way  in  which  the  tasks 
required  by  a  user  are  executed.  This  arises  with 
the  parallel  use  of  processors  and  possible  restric¬ 
tions  on  how  much  parallelism  the  algorithm  or 
the  application  allows.  It  is  on  this  type  of  prob¬ 
lem  that  we  focus  in  the  remainder  of  this  paper. 
Below,  we  give  a  description  of  the  problem,  a 
model  of  the  system,  analysis  of  its  behavior  and  a 
discussion  of  the  results. 


3.  A  Model  for  Synchronization  Beween  Two 
Processes 

Assume  we  have  a  job  which  is  partitioned  into 
two  processes,  each  of  which  is  executed  on  a 
separate  processor.  As  these  processes  are  ex¬ 
ecuted,  we  consider  that  they  advance  along  the 
x-axis  in  steps  of  length  one  (i.e.  they  visit  the 
non-negative  integers),  each  beginning  at  x  =  0  at 
time  t  =  0.  Each  process  independently  takes  m 
exponentially  distributed  amount  of  time,  with 
parameter  X,  (<  =  1,  2),  to  advance  from  position 
k  to  position  /c  -F  1  (X  =  0,  1,  2, . . . ).  When  pro¬ 
cess  i  advances  one  unit  along  this  axis,  it  will 
send  a  message  to  the  other  process  with  probabil¬ 
ity  (0  <  Qi  <  1).  Upon  receiving  a  message  from 
the  other  (sending)  process,  this  (receiving)  pro¬ 
cess  wUl  do  the  following: 

(1)  If  its  position  along  the  x-axis  is  equal  to  or 
behind  the  sending  process,  it  will  ignore  the  mes¬ 
sage. 

(2)  If  it  is  ahead  of  the  sending  process,  it  will 
inunediately  move  back  (i.e.,  “rollback”)  along  the 
x-axis  to  the  current  position  of  the  sending  pro¬ 
cess. 

This  is  a  simple  model  of  distributed  simulation 
(motivated  by  the  time  warp  distributed  simula¬ 
tion  algorithm  [S])  where  two  processors  are  both 
working  on  a  simulation  job  in  an  effort  to  speed 
it  up.  They  both  proceed  independently  until  such 
time  as  one  (slower)  process  transmits  a  message 
in  the  “past”  of  the  other  (faster)  process.  This 
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causes  the  faster  process  to  “rollback”  to  the 
point  that  the  slower  process  is  at,  after  which 
they  advance  independently  again  until  the  next 
rollback,  etc. 

Let  F(t)  =  position  of  the  first  process  (process 
1)  at  time  /  and  let  5(/)  =  position  of  the  second 
process  (process  2)  at  time  t.  Further,  let 

Z)(r)  =  F(/)-S(/); 

D(t)  =  0  whenever  (2)  occurs  (i.e.,  a  rollback).  We 
are  interested  in  studying  the  Markov  process 
D(t).  From  our  assumptions  that  F(0)  =  5(0)  =  0, 
we  have  Z)(0)  =  0.  Clearly,  D(t)  can  take  on  any 
integer  value  (i.e.,  it  certainly  can  go  negative).  We 
will  solve  for 

Pk^  Urn  P[D{t)=k], 

t-*  00 

k=  ...,  -2,  -1,0,  1,2,...  (3.1) 

namely,  the  equilibrium  probability  for  the 
Markov  Chain  D(t).  Moreover,  we  will  find  the 
average  separation  between  processes  as  well  as 
the  speedup  with  which  the  computation  proceeds 
when  using  two  processors  relative  to  the  use  of  a 
single  processor  as  described  below. 

Our  model  is  that  of  a  discrete  state,  continu¬ 
ous  time  process.  Some  previous  work  on  varia¬ 
tions  of  this  model  already  exists.  For  example, 
Mitra  and  Mitrani  [6]  studied  a  related  model  in 
which  they  considered  a  continuous  state,  discrete 
time  model  not  unlike  the  one  we  have  described 
here.  Their  results  are  similar  to  ours  in  some  ways 
although  their  method  of  analysis  appears  to  be 
far  more  complex  than  ours.  Lavenberg,  Muntz, 
and  Samadi  [7]  provide  an  approximate  analysis 
of  a  continuous  time,  continuous  state  model.  (We 
have  also  completed  the  analysis  of  the  discrete 
time,  discrete  state  case  which  will  be  published 
elsewher.)  Moreover,  Feldennan  and  Kleinrock  [8] 
give  an  upper  bound  on  the  gain  in  speedup  that 
P  unsynchronized  processors  can  achieve  relative 
to  P  processors  which  are  forced  to  synchronize  at 
every  step. 


4.  Amdysis 

Let  us  analyze  the  behavior  of  these  two  cou¬ 
pled  processes  which  we  have  modeled  as  a  one¬ 
dimensional  discrete  state,  continuous  time 


Markov  Chain.  The  following  balance  equations 
are  easily  obtained: 


iK 

+  ^2)P* 

—  P*  -  1  ^  2<72  P*  +  1  ’ 

k 

=  1,2,.. 

.  , 

(4.1) 

+ 

1 

k  ~^2P-(*-l)‘*‘  ^l9lP-(*  +  l)’ 

k 

=  1,2,.. 

•  < 

(4.2) 

^  2 )  Po 

30  00 

=  ^2^2  L  P*  +  ^l9l  E  P-k 

k~l  k-\ 

+  ^2^2  Pi  +  ^1^1  P-1 

(4.3) 

where  q,  =  I  -  q,. 

We  solve  this  system  of  linear  difference  equa¬ 
tions  using  the  usual  approach  of  z-transforms  [1] 
by  defining  the  two  transforms 

(4.4) 

Q(2)=LP-k^*-  (4.5) 


We  further  define  the  relative  speed  parameter 
^  “  Xj  +  Aj  ■ 


Multiplying  (4.1)  by  r*  and  summing  over  the 
range  k  =  1,  2, ...  we  obtain 


R(z) 


ziofllPl-OPoZ) 
az^  —  z  +  aqi 


(4.7) 


where  a  =  1  —  a.  Similarly,  multiplying  (4.2)  by  z* 
and  summing  over  the  range  k  —  1,  2, ...  we  obtain 


QU) 


^{a<i\P-\-aPo2) 
ciz^  —  z  -f  aq-i 


(4.8) 


Note  the  duality  between  R{z)  and  Q{z)  with 
regard  to  the  variables  a,  and  92- 

The  denominator  roots  (i.e.  the  poles)  of  R{z) 
are  given  by 
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It  is  easy  to  show  that  these  four  poles  are  real 
and  that  0  <  rj  <  l,  i  <  r^.  0  <  Sj  <  1  and  1  <  Sj. 
Since  R{z)  and  Q{z)  are  both  analytic  in  the 
region  |  z  [  <  1,  it  must  be  that  numerator  (/?(/•,)) 
=  0  and  numerator  (0(j,))  =  O.  From  this  ob¬ 
servation  and  from  (4.7)  and  (4.8)  we  have 


P\ 


aq-^P^' 


(4.11) 


as, 


(4.12) 


In  addition,  by  conserving  probability  we  have 
/>o  +  /?(l)  +  e(l)  =  l.  (4.13) 

From  these  last  three  we  readily  find 


Po  = 


{ri  -  I)(^2  -  I) 

•"iSi  -  1 


(4.14) 


From  (4.11)  and  (4.12)  we  may  simply  rewrite 
R{z)  and  Q{z)  as 


R{z) 


Q{z) 


ZPo 

S2-Z- 


(4.15) 


We  may  now  invert  both  R{z)  and  Q(z)  to 
give  us  the  equilibrium  distribution  for  our  Markov 
Chain,  namely. 


A:  =  0,1.2 . 

*  =  0,  -1,  -2 . 


(4.16) 


Equation  (4.16),  along  with  (4.9),  (4.10)  and  (4.14), 
give  us  the  complete  solution  to  the  Markov  Chain. 

To  find  the  average  separation  between  these 
two  processes  on  the  x-axis,  namely,  K  = 
lim,_„£[  I  D{t)  |],  we  calculate  as  follows; 

£  k{p^+p_^) 

k-l 


which  gives  us 


f^-Po 


'2 


+ 


^2 

(^2-1)' 


(4.17) 


Let  us  now  calculate  the  speedup  5  which  is 
defined  as  the  rate  at  which  the  two  processor 
system  carries  out  useful  processing  divided  by  the 
rate  at  which  an  equivalent  single  processor  carries 
out  useful  processing.  We  define  the  equivalent 
single  processor  as  one  which  moves  a  process 
along  the  time  axis  at  a  rate  equal  to  the  average 


rate  of  the  two  original  processes,  namely:  ifX,  + 
Xj).  In  this  single  processor  case,  there  is  no 
rollback  to  worry  about  and  so  useful  progress 
occurs  at  the  rate  3(X,  +  Xj).  In  the  two  processor 
case,  useful  progress  is  equal  to  the  sum  of  the  two 
rates  minus  the  expected  rollback  for  each  pro¬ 
cess.  If  D(t)  =  k  at  time  /.  and  if  the  next  advance 
along  the  x-axis  is  made  by  the  lagging  process 
which  also  causes  the  leading  process  to  rollback 
(with  probability  qj ),  then  the  leading  process  will 
be  rolled  back  only  a  distance  X  -  1  since  the 
lagging  process  just  advanced  one  step  along  the 
x-axis.  Thus,  we  see  that  the  rate  at  which  the  two 
processor  system  advances,  on  average,  is  given  by 

X,  +  X2  —  £  ^iqiPk^k  —  1) 

/t-i 

ao 

-  L  Kq\P-Ak-\). 

k~\ 


Thus,  the  speedup  is  given  by 
S  =  2-2aq2  £  p*(X-l) 

£  p_*(X-  1). 


This  leads  us  to  the  following  general  expression 
for  the  system  speedup: 


^qiPo _ ^q\Po 

(.2-1)^ 


(4.18) 


5.  The  Symmetric  Cstse 

In  this  section  we  consider  the  symmetric  case 
where  q\  =  qz  —  q\  that  is,  each  process  has  the 
same  probability  of  sending  a  message  to  the  other 
process.  We  now  have  ar^  =  or,  and  arj  =  otj- 

The  speedup  is  shown  in  Fig.  3  as  a  function  of 
a  and  q.  For  a  =  the  speedup  rises  continously 
to  its  maximum  value  of  5*2  as  ^  0.  For 

9*0,  5  *  2  for  all  a  but  5  has  a  di^ntinuity 
for  aU  this  discontinuity  is  not  shown 

clearly  in  Fig.  3.  (For  9  =  0,  no  rollbacks  occur 
and  it  is  intuitively  clear  that  5*2.)  Note  for 
9  >  0  that,  when  a  -» 0  or  a  -» 1  (that  is,  X,  -» 0 
or  X2  -» 0),  then  the  speediip  goes  to  0;  this  is  the 
case,  since  one  process  moves  extremely  slowly 
(compared  to  the  equivalent  single  process)  and  it 
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will  occasionally  drag  the  faster  process  back  to  its 
lagging  position. 


6.  The  Balanced  Case  X,  =  Xj 


Here  we  consider  the  balanced  case  where  both 
processes  move  at  the  same  rate  giving  us  a  =  (1 
-  a)  =  j.  We  see  that  r^  =  l  -  rj  =  1  + 

■^1  ~  1  ~  5;  =  1  +  In  addition, 

_ il'li  ,  , 

Furthermore,  in  this  case,  we  find  that  the  speed¬ 
up  is  simply 


S  = 


2(v^  +  {Ti) 

+  ' 


(6.2) 


In  Fig.  4  we  show  the  speedup  as  a  function  of  qx 
and  q^.  Note,  of  course,  that  the  speedup  goes  to  2 
for  qx-q2  =  ^  and  goes  to  |  for  =  ^2  =  1- 


Fig.  4.  Speedup  for  the  balanced  case,  X]  •>  X2  (a  ••  2). 


7.  The  Symmetric  Balanced  Case:  qx  —  q^  and  X] 
-X2 


In  this  symmetric  balanced  case,  where  both 
processes  move  at  the  same  speed  and  both  have 
the  same  probability,  qx=  qi^  of  passing  a 
message  to  the  other  process,  we  obtain  great 
simplifications.  In  particular,  we  have  r,  =  =  1 

-  yjq  and  ^2  =  =  1  +  Note,  again,  that  a  = 

(1  —  a)  =  2.  In  this  case  we  find. 


/>o  = 


1  +  xfq 


(7.1) 


The  average  process  separation  becomes 

r,  2(1  +  fq^ 

v/^(2  + v^)  ■ 


(7.2) 


The  speedup  is  given  by 


S  = 


4 

1  +  yfq' 


(7.3) 


The  speedup  function  in  this  very  special  case  is 
shown  in  Fig.  5.  Note  for  9  =  0  that  5  =  2  whereas 
for  ^=1  we  have  5=|.  We  can  see  this  last 
result  intuitively  as  follows.  If  each  process  always 
sends  a  message  to  the  other  process  when  it 
advances,  then  the  time  for  both  processes  to 
advance  one  unit  is  equal  to  the  maximum  of  two 
exponential  delays  which  we  know  is  equal  to  1.5 
times  the  mean.  Thus,  the  rate  of  progress  for  each 
process  is  simply  f  times  the  rate  of  a  single 
process.  Since  both  are  moving  at  a  rate  f,  the 
sum  is  equal  to  |  which  yields  the  result  for  q  =  \. 


I 


% 


Fig.  S.  Speedup  for  the  symmetric,  balanced  case,  ■■  ~  9. 

X1-X2  (a-j). 
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8.  Condusions 

In  this  paper  we  have  focused  on  the  speedup 
available  when  two  processes  limit  each  others’ 
rate  of  progress  by  passing  messages  between  each 
other  (these  messages  may  cause  a  process  to  roll 
back  and  therefore  waste  some  of  the  work  it  has 
accompUshed).  We  gave  the  complete  solution  for 
the  important  system  performance  variables  in  the 
general  case.  We  found  for  the  symmetric  bal¬ 
anced  case  that  when  ^  =  0  the  speedup  reaches  its 
maximum  value  of  2  but  that  the  speedup  falls  off 
infinitely  quickly  as  q  increases  from  0,  the  limit¬ 
ing  speedup  being  ^  at  ^  =  1. 

The  analysis  given  here  is  for  only  two 
processors.  For  larger  numbers  of  interacting 
processes,  the  mathematics  becomes  far  more 
complex  and  the  most  likely  fruitful  approach  to 
understand  the  interaction  among  many  asynch¬ 
ronous  processes  would  be  via  approximate  solu¬ 
tions. 

Synchronization  is  only  one  source  of  overhead 
one  finds  in  distributed  systems.  There  are  numer¬ 
ous  other  issues  which  impact  the  performance  of 
distributed  systems.  Indeed,  we  are  only  beginning 
to  formulate  the  models  and  gain  the  understand¬ 
ing  required  to  analyze,  design,  and  properly  im¬ 


plement  distributed  systems  in  this  coming  de¬ 
cade. 
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ISDN — The  Path  to  Broadband  Networks 


LEONARD  KLEINROCK,  fellow,  ieee 

Invited  Paper 


We  are  in  the  midst  of  revolutionary  improvements  in  data  communi¬ 
cations.  The  need  for  connectivity  has  never  been  as  great  as  it  is  today 
due  to  the  rapid  growth  of  desktop  processing  machines  which  must  com¬ 
municate  among  themselves  as  well  as  with  centralized  computing  and 
database  facilities.  Alas,  in  the  midst  of  this  progress,  we  find  ourselves 
burdened  by  the  curse  of  incompatibility  among  vendor-specific  products, 
protocols,  procedures,  and  interfaces. 

At  the  same  time,  the  natioruil  and  international  bodies  have  been  hard 
at  work  attempting  to  provide  some  stability  by  introducing  standards  for 
connectivity.  The  problem,  of  course,  is  one  of  timing;  a  premature  stan¬ 
dard  stifles  the  development  of  mature  technology,  while  a  tardy  standard 
is  in  danger  of  being  rejected  by  a  community  that  is  locked  into  irrever¬ 
sible  commitments  to  cumbersome  ad  hoc  solutions.  ISDN  is  an  emerging 
standard  which  represents  an  international  effort  to  solve  some  of  our 
connectivity  problems.  If  it  rolls  out  in  a  timely  fashion  and  addresses 
real  needs  to  the  end  user  community,  it  has  a  chance  for  success  in  the 
networking  world. 

The  carriers  are  committed  to  ISDN  and  have  a  clear  motivation  and 
potential  for  succeeding  in  its  development.  Narrowband  ISDN  is  a  ho- 
hum  service  for  which  some  important  applications  have  been  identified, 
but  which  has  not  sparked  a  stampede  of  acceptance.  On  the  other  hand, 
broadband  ISDN  (BISDN)  is  a  service  that  has  identified  capabilities  that 
are  truly  exciting  and  could  very  well  dominate  data  networking  in  this 
decade.  The  success  of  BISDN  will  depend  strongly  on  the  rollout  of  prod¬ 
ucts.  the  ubiquity  of  its  presence,  and  the  tarriffing  of  its  services. 

I.  Introduction 

Telecommunications  is  currently  a  huge  industry  approaching 
an  annual  revenue  of  $200  000  000  000;  it  has  one  of  the  fastest 
growth  rates  of  all  industries  today.  Moreover,  it  is  based  on  some 
of  the  most  exciting  technologies  available,  changing  rapidly,  and 
influencing  almost  every  aspect  of  business,  commerce,  educa¬ 
tion,  health,  government,  and  entertainment.  Its  products  are  vis¬ 
ible  to  everyone,  and  yet,  the  full  impact  of  this  juggernaut  is  not 
yet  appreciated  by  most  observers. 

What  has  caused  this  enormous  growth  has  been  the  explosion 
of  digital  technology  (which  itself  was  fueled  by  semiconductor 
electronics,  namely,  integrated  circuits  of  very  large  scale,  as  well 
as  the  development  of  the  unbelievable  capabilities  of  fiber-optic 
communication).  This  digital  technology  appeared  first  as  data- 
processing  machines  and  soon  had  its  impact  on  data  communi¬ 
cations.  This  impact  emerged  as  data  communication  networks, 
principally  in  the  form  of  packet  switching  in  the  1970’s  {!].  Since 
then,  the  dau-processing  industry  and  the  data  communication 
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industry  have  converged  in  a  fashion  that  will  never  again  let 
them  separate.  You  can  no  longer  discuss  one  without  the  other. 

The  product  rollout  has  been  staggering  and  we  have  been  pro¬ 
vided  a  broad  range  of  advanced  services,  but  not  without  a  price. 
We  have  now  reached  a  stage  of  uncontrolled  chaos  in  the  mar¬ 
ketplace  of  data  processing  and  data  communications.  Multiven¬ 
dor  systems  are  almost  universal,  and  the  inability  of  the  elements 
in  this  heterogeneous  environment  to  interwork  is  legion.  There 
have  been  international  efforts  to  bring  some  order  to  this  chaos 
through  the  introduction  of  staitdards.  Such  efforts  are  almost 
always  slow,  laborious,  political,  petty,  boring,  ponderous, 
thankless,  and  of  the  utmost  criticality.  The  International  Stan¬ 
dards  Organization  has  developed  the  seven-layer  Open  Systems 
Interconnection  (OSI)  reference  model  for  communications.  The 
IEEE  802.x  series  of  standards  for  communications  is  growing. 
We  have  seen  the  Consultative  Committee  for  International 
Telephony  and  Telegraphy  (CCITT)  recommendations  for  their 
X  series  of  standards  proliferate.  Moreover,  and  of  most  interest 
to  this  paper,  CCITT  has  been  developing  the  Integrated  Services 
Digital  Network  (ISDN)  standard  since  the  mid-1970’s.  The  def¬ 
inition  and  details  of  this  standard  are  coveted  elsewhere  in  these 
Proceedings. 

It  is  the  purpose  of  this  paper  to  evaluate  the  effect  of  ISDN  on 
the  field  of  data  networks,  to  anticipate  future  directions  for  this 
technology,  and  to  discuss  how  the  user  should  view  these  devel¬ 
opments. 

Whereas  this  paper  discusses  such  issues,  the  fact  is  that  the 
underlying  issue  is  really  one  of  infrastructure,  rather  than  of 
ISDN  networking  by  itself.  Network  technology  provides  us  the 
capability  to  insttll  a  powerful  communications  and  information 
technology  infirastiuctuie  that  will  enable  untold  growth  and  access 
in  the  years  to  come.  ISDN  is  one  cornerstone  of  that  technology. 

n.  Current  Status 

There  are  more  than  200  000  ISDN  access  lines  installed  today, 
and  that  number  will  likely  grow  to  three-quarters  of  a  billion  by 
199S  [2].  Its  use  in  public  networks  is  clear,  and  it  is  beginning 
to  penetrate  the  private  network  market  as  well.  It  has  taken  29 
years  from  the  first  digital  T1  system  to  today's  ISDN  develop¬ 
ments.  1988  was  a  critical  year,  for  it  was  in  that  year  that  Sig¬ 
nalling  System  Seven  (SS7)  installations  increased  enormously, 
providing  the  out-of-band  common  channel  signaling  capability 
on  which  ISDN  is  based  [3].  We  have  seen  a  very  rapid  rollout 
from  the  availability  of  the  bask  rate  interfile  (BRI)  at  144  kb/s 
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(2  B  -t-  D)  and  the  primaiy  ntt  iateifKX  (PRI)  at  23  x  64  kb/s 
to  today’s  bfgTnning  of  BISDN  at  155  Mb/s  and  growing  to  over 
13  Gb/s  speeds.  Indeed,  we  have  already  seen  the  early  dem¬ 
onstrations  of  the  802.6  Metropolitan  Area  Network  (MAN)  stan¬ 
dard  based  on  the  distributed  queue  dual  bus  (DQDB)  access 
method;  this  demonstation  was  pan  of  the  switched  multimegabit 
data  service  (SMDS)  offered  at  45  Mb/s.  Things  are  moving 
quickly. 

A.  The  Barriers 

Indeed,  it  is  remarkable  that  ISDN  is  here  at  all,  given  the  large 
number  of  compelling  barriers  that  it  has  had  to  overcome.  Pri¬ 
marily,  the  problem  has  been  that  ISDN  is  a  technology  devel- 
oped  and  desired  by  the  carriers,  and  not  one  that  was  initiated 
by  user  demand.  As  a  result,  a  deadlock  persisted  that  took  the 
following  form.  First,  the  carriers  were  unwilling  to  deploy  a  cen¬ 
tral  office  ISDN  switch  until  they  could  estimate  the  market  that 
would  justify  the  huge  expenditures  involved.  The  market  could 
not  be  estimated  until  the  users  Judged  their  likely  use  of  the  tech¬ 
nology;  but  the  users  could  not  make  this  judgement  until  they 
could  be  given  cost  and  timing  of  the  ISDN  products.  To  provide 
this  product  cost  and  timing  information,  the  system  suppliers 
needed  the  chip  set  cost.  But  the  chip  nunufacturers  were  unwill¬ 
ing  to  tool  up  until  they  could  see  the  nuitket  that  could  not 
develop  until  the  central  office  switches  were  in  place.  This  dead¬ 
lock  could  only  be  broken  by  the  carriers  who  did  indeed  take  the 
first  step  and  got  the  process  moving. 

As  we  unwind  from  this  deadlock,  users  are  concerned  that  if 
they  buy  now  and  ISDN  is  a  failure,  then  they  will  be  left  stranded 
with  an  obsolete  technology  whereas  if  ISDN  is  successful,  then 
costs  will  drop  due  to  the  usual  economies  of  scale.  In  both  cases, 
the  user  is  motivated  to  wait;  the  user  is  clearly  unclear  as  to  when 
he  should  jump  on  the  ISDN  bandwagon.  Further,  the  real  attrac¬ 
tion  of  ISDN  will  come  when  the  service  is  ubiquitous  and 
becomes  available  in  all  of  the  locations  in  which  he  is  interested; 
but  networking  technology  expands  at  a  slow  rate  due  largely  to 
the  enormous  cost  of  providing  broad  coverage.  We  have  seen 
this  curse  of  distributed  services  many  times  in  the  past;  for 
example,  it  occurred  with  the  introduction  of  telephones,  of  Fed¬ 
eral  Express  overnight  mail,  of  public  packet  switched  networks, 
of  FAX,  of  electronic  mail,  and  more. 

The  problem  is  further  exacerbated  by  the  fact  that  not  all 
implementations  of  ISDN  products  are  interoperable;  for  exam¬ 
ple.  it  is  the  usual  case  that  ISDN  adapters  from  different  manu¬ 
facturers  cannot  communicate  with  each  other.  The  average  price 
of  an  ISDN  adapter  for  a  PC  today  is  $1500,  whereas  adapters 
for  LAN  interconnection  of  PC’s  sell  for  less  than  $800  (and 
include  a  microprocessor  as  well).  The  full  ISDN  standard  has 
not  yet  been  finalized  by  the  CCITT.  The  fact  that  there  is  no 
equivalent  of  the  Corporation  for  Open  Systems  (COS)  for  ISDN 
leads  to  the  problem  of  vendor  products  that  are  incompatible. 
The  existence  of  more  than  one  version  of  a  standard  is  an  oxy¬ 
moron.  And  the  specter  of  possible  changes  in  the  standard  or  in 
the  unofficial  portions  of  the  standard  many  well  cause  today's 
purchased  equipment  to  become  obsolete. 

B.  The  Enablers 

In  spite  of  the  barriers  seen  by  the  carriers,  the  suppliers,  and 
the  users  to  the  introduction  and  deployment  of  ISDN,  these  same 
groups  see  significant  advantages  to  ISDN  that  have  been  hasten¬ 
ing  its  introduction. 


The  carriers  have  passed  through  a  number  of  years  of  equal 
access  since  divestiture,  which  has  produced  a  highly  competitive 
marketplace.  They  have  been  energized  to  offer  more  than  just 
transport  and  to  extend  their  offerings  to  central-office  based  ser¬ 
vices  of  various  types,  most  of  which  are  dependent  upon  the 
introduction  of  ISDN.  Moreover,  the  flattening  demand  of  PBX 
equipment  has  produced  a  marketplace  in  which  one  vendor’s  gain 
is  the  other  vendor’s  loss  (i.e.,  a  zero-sum  game).  Consequently, 
a  carrier  must  add  value  to  its  offerings  to  differentiate  it  and  to 
expand  the  size  of  its  market;  ISDN  is  the  vehicle  for  this  added 
value.  The  chip  nunufacturers  have  long  since  recognized  that 
the  iiuss-produced  memory  chip  nurketplace  has  been  lost  to  the 
Japanese.  These  nunufacturers  need  other  nurkets,  and  the  ISDN 
chip  iturket  is  an  attractive  one  for  them. 

Major  corporate  users  have  seen  the  cost  of  their  separate  voice 
and  data  networks  rise.  These  users  have  begun  to  recognize  that 
an  advanced,  integrated  corporate  network  offers  them  a  critical 
competitive  edge  as  well  as  lower  network  costs.  The  additional 
function  being  offered  by  advanced  networks  is  becoming  very 
attractive  to  them  and  their  top  management  is  being  convinced 
of  these  facts.  ISDN  offers  a  migration  path  to  achieve  these  goals. 
The  first  customers  of  the  ISDN  services  have  been  very  large 
organizations  with  growing  networking  needs;  the  large  con¬ 
sumer  contact  firms  (e.g.,  American  Express)  are  quickly  moving 
in  this  direction. 

The  success  of  ISDN  depends  critically  upon  the  success  of  the 
applications  that  take  advantage  of  its  capabilities.  Indeed,  it  is 
the  identification  and  development  of  a  rich  set  of  applications 
that  will  hasten  the  growth  of  ISDN  more  than  any  other  factor. 
We  have  seen  this  phenomenon  at  work  in  a  number  of  other 
network  related  systems  in  the  past.  Packet  switching  succeeded 
in  the  commercial  environment  largely  because  of  the  electronic 
mail  application  that  it  supported.  SNA  took  hold  because  of  the 
support  it  provided  for  transaction  processing.  PC  LAN's  have 
proliferated  because  of  the  need  to  share  peripherals  and  data. 

We  have  yet  to  identify  the  hot  new  application(s)  that  will 
drive  ISDN  steeply  up  the  demand  curve.  Some  of  the  applica¬ 
tions  that  have  oeen  identified  so  far  include  automatic  number 
identification  (ANI)  as  well  as  the  ability  to  turn  off  ANI,  reduced 
call  setup  time  from  20  s  to  less  than  3  s.  the  availability  of  a 
single  access  point  for  digital  services  (thus  eliminating  multiple 
dedicated  access  lines),  the  ability  to  provide  video-based  tele¬ 
phony.  voice-data  applications,  desktop  ISDN  links,  etc.  So  far. 
none  of  these  have  sparked  a  rush  to  the  ISDN  market. 

Nevertheless,  the  carriers  are  overwhelmingly  behind  ISDN  and 
they  will  do  all  in  their  power  to  promote  it.  It  is  in  their  interest 
to  do  so.  In  the  long  run.  it  will  be  in  the  user's  interest  as  well, 
for  the  carriers  are  the  ones  who  will  provide  the  networking 
infrastructure  that  is  called  for.  Today's  networks  are  disorga¬ 
nized.  expensive,  not  integrated,  slow,  complex,  difficult  to  man¬ 
age.  and  unable  to  interoperate  with  each  other;  an  international 
standard  interface  such  as  ISDN  is  badly  needed.  To  their  credit, 
the  Europeans  have  been  much  more  aggressive  than  the  North 
Americans  in  implementing  ISDN.  And  if  you  still  doubt  that  the 
case  for  ISDN  is  justified,  consider  the  fact  that  the  less-devel¬ 
oped  and  under-developed  regions  of  the  world  are  anxious  to 
connect  to  the  world  standard  network.  There  is  no  way  that  each 
of  them  can  or  should  establish  their  own  standard.  There  abso¬ 
lutely  must  be  an  available  world  standard  to  which  they  can 
attach. 

ISDN  is  a  technology  that  allows  those  who  have  not  kept  pace 
with  the  growth  in  networking  technology  to  catch  up  immedi¬ 
ately. 
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III.  Narrowband  ISDN  is  not  Enough 

The  BRI  and  PRI  ISDN  offerings  are  often  collectively  referred 
to  as  narrowband  ISDN  (NISDN)  to  distinguish  them  from 
BISDN.  The  data  rates  associated  with  NISDN  are  inadequate  for 
many  applications  of  interest.  On  the  one  hand,  the  BRI  providing 
64  kb/s  channels  is  not  a  large  improvement  over  today's 
modems,  which  provide  data  service  at  9.6  kb/s  and  19.3  kb/s 
and  which  are  widely  available.  It  is  also  the  case  that  64  kb/s 
is  a  nonstarter  for  the  data  transmission  speeds  to  which  today's 
users  have  become  accustomed  (e.g.,  local  area  networks  running 
at  10  Mb/s  and  more).  The  PRI  tunning  at  1.54  Mb/s  is  a  clear 
improvement  over  BRI,  but  is  no  different  in  available  speed  than 
is  the  popular  T1  offerings  in  use  by  the  community  today  (so 
why  abandon  T1  and  introduce  new  equipment  interfaces  for 
PRI?).  Add  to  that  the  nasty  incompatibilities  faced  by  multina¬ 
tional  corporations  when  they  find  that  PRI  in  Europe  is  2.05 
Mb/s  rather  than  1.54  Mb/s  in  North  America;  of  course,  this 
problem  already  exists  in  today's  T1  offerings.  The  PRI  rate  is 
still  a  significant  step  away  from  the  bandwidth  needs  of  the  data 
processing  community:  it  takes  almost  5  min  to  move  a 
50-megabyte  file  at  TI  transmission  speeds. 

From  the  viewpoint  of  data  networks,  the  real  excitement  of 
ISDN  comes  about  when  one  discusses  the  capabilities  of  BISDN. 
155  Mb/s  is  a  real  improvement  over  today's  speeds.  The 
50-megabyte  file  can  now  be  moved  in  2.5  s!  The  precursor  to 
BISDN  is  the  growing  use  of  the  T3  service  (45  Mb/s).  Indeed, 
the  huge  popularity  of  TI  and  the  growing  popularity  of  T3  are 
setting  the  stage  for  the  introduction  of  BISDN  at  155  Mb/s  and 
620  Mb/s. 

The  need  for  broadband  speeds  comes  from  a  number  of  appli¬ 
cations.  The  existence  of  today's  high  bandwidth  customer  prem¬ 
ises  networks  (i.e.,  local  area  networks  (LAN's)  require  long  dis- 
unce  broadband  to  interconnect  them;  LAN  interconnetion  using 
switched  broadband  data  service  is  a  clear  and  current  applica¬ 
tion.  The  emerging  field  of  teleradiology  in  which  one  transmits 
medical  imagery  among  hospitals,  physicians,  and  patients 
requires  large  bandwidths  due  to  the  enormous  data  files;  the  typ¬ 
ical  pair  of  chest  X-rays  we  all  get  in  a  routine  medical  exami¬ 
nation  requires  as  much  storage  as  four  volumes  of  the  Encyclo¬ 
pedia  Britannica.  A  similar  need  comes  from  the  field  of 
telepathology,  i.e.,  the  transmission  of  optical  images  of  biolog¬ 
ical  samples.  On-line  access  to  supercomputer  output  showing 
real-time  rotation  of  complex  molecules  in  three  dimensions  in 
full  color  can  be  a  real  bandwidth  hog.  File  server  access  to  rapid 
scanning  of  visual  and  textual  data  is  another  application.  The 
growth  of  CAD  applications  will  be  one  source  of  rapid  devel¬ 
opment  and  deployment  of  customized  ISDN  chips. 

Itideed,  the  first  applications  of  BISDN  will  be  in  the  commer¬ 
cial  and  scientific  sectors.  However,  following  that,  a  real  drive 
for  broadband  will  be  in  the  residential  sector  in  order  to  provide 
entertainment.  For  example,  CATV  cable  service  passes  by  86% 
of  American  homes,  55%  of  homes  subscribe  to  CATV  cable 
services,  30%  of  homes  purchase  more  than  one  premium  movie 
channel.  10%  buy  pay-per-view  services,  and  the  average  home 
consumes  7  hours  of  television  per  day  [4].  HDTV  will  increase 
the  demand  for  sevices  and  will  place  enormous  bandwidth 
requirements  on  our  communication  plant.  If  the  FCC  allows 
CATV  services  to  be  offered  by  the  telephone  companies,  it  would 
be  a  tremendous  pull  from  the  demand  side  for  the  installation  of 
broadband  capability  to  the  subscriber  base.  Of  course,  optical 
fiber  will  be  the  medium  providing  these  large  bandwidths.  and 
the  economies  that  suppoit  fiber  installation  are  already  here. 


Currently,  there  are  well  over  a  million  miles  of  installed  fiber  in 
the  U.S.  It  is  now  less  expensive  to  install  fiber  than  it  is  to  install 
copper  for  large  office  buildings.  Fiber  to  the  curb  (FTTC)  is 
becoming  competitive  for  new  installations,  and  fiber  to  the  home 
(FTTH)  is  under  serious  consideration  already.  The  appropriate 
strategy  is  to  begin  the  FTTC  and  FTTH  installations  now.  while 
NISDN  is  deploying. 

Thus  the  real  payoff  in  the  data  networks  world  for  ISDN  is  the 
promise  of  BISDN  and  all  the  services  and  capabilities  it  will 
bring. 

rv.  Current  Nets  are  Inadequate 

It  is  clear  that  the  data  networks  we  inherited  from  the  1980’s 
are  inadequate  to  handle  the  applications  and  capabilities  required 
oy  the  1990's.  Today's  packet  switched  data  networks  have  a 
number  of  problems  with  them:  They  are  high  cost,  they  are  low 
speed,  they  introduce  large  switching  delays,  they  have  relatively 
high  error  rates,  the  switches  require  too  much  intelligence,  the 
switches  are  electronic,  there  is  too  much  storage  in  the  network, 
the  protocols  are  too  heavyweight,  and  too  much  processing  is 
done  in  the  network. 

For  example,  X.25  packet  switching  networks  are  serving  a 
real  need  as  they  currently  exist.  However,  they  are  based  essen¬ 
tially  on  64-kb/s  speeds  and  use  heavyweight  protocols  (they 
process  up  to  layer  3  at  every  hop).  As  an  alternate  to  X.25  packet 
switching,  frame  relay  is  currently  being  considered  for  the  interim 
version  of  fast  packet  switching,  whereby  the  LAPD  link  level 
protocol  will  be  used  to  perform  switching  functions  at  layer  2 
without  the  layer  3  processing  overhead  [5],  [6]. 

Tomorrow’s  broadband  networks  require  new  architectures  to 
handle  the  changing  requirements.  The  move  from  megabits  per 
second  to  gigabits  per  second  requires  dramatic  changes  in  think¬ 
ing  and  in  stracture.  In  Table  1  we  list  some  of  these  contrasts. 


Table  1  Packet  Network  Characteristics:  Present  Versus  Fumre 


Today 

Broadband 

Packers /s 

Thousands 

Millions 

Bandwidth 

64  kb/s 

150  M/-620  Mb/s 

Bandwidth  allocation 

Fixed 

Dynamic 

Services 

Voice,  data 

Integrated  voice, 
data  and  image 

Switch  delay 

50-l(X)  ms 

10  ms 

Propagation  delay 

Insignificant 

Dominant 

Error  control 

Link-to-link 

End-to-end 

Protocols 

Heavyweight 

Lightweight 

Bottleneck 

Link  bandwidth 

Switch  bandwidth 

The  path  from  today's  data  networks  to  those  of  tomorrow  is 
being  paved  right  now.  T3  offerings  at  44.7  Mb/s  are  beginning 
to  penetrate  the  private  networking  marketplace.  The  synchron¬ 
ous  optical  network  (SONET)  standard  for  optical  transmission 
was  agreed  upon  by  the  CCITT  in  1988  [7]  and  has  promoted 
BISDN  product  development.  The  operations,  administration,  and 
maintenaiKe  (OA&M)  portions  of  the  SONET  standard  should 
be  completed  by  the  end  of  1990  and  will  only  require  software 
updates  to  implement.  SONET  has  laid  out  a  hierarchy  of  trans¬ 
mission  speeds  from  51.8  Mb/s  up  to  13.27  Gb/s  and  higher. 
These  enormous  speeds  are  fine  for  point  to  point  communica¬ 
tions  (assuming  the  end  points  can  gobble  up  gigabits  per  sec¬ 
ond),  but  certainly  place  some  outrageous  demands  on  the  inter¬ 
nal  switches  in  the  network. 

These  very  large  communication  bandwidths  have  caused  a 
wealth  of  research  and  experimentation  to  take  place  in  the 
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leseuch  laboratories  in  the  advanced  area  of  fast  packet  switching 
(8).  Fast  packet  switching  will  likely  use  parallel  i»ocessing 
architectures  in  the  switch  to  handle  the  millions  of  packets  per 
second  mentioned  above.  There  is  a  number  of  competing  archi¬ 
tectures  being  proposed  for  the  interconnection  netwoiks  within 
these  switchra  and  many  of  them  use  the  Banyan  switch  in  one 
fonn  or  another  [9].  The  advantage  of  these  architectures  is  that 
many  packets  can  be  switched  simultaneously  through  the  switch¬ 
ing  fabric  using  the  concurrent  processing  capability  of  the  par¬ 
allel  processors. 

A  new  multiplexing  scheme  known  as  asynchronous  transfer 
nxxle  (ATM)  [10]  has  been  adopted  for  BISDN  which  uses  fixed 
length  packets  (called  cells)  of  length  S3  bytes  (48  bytes  of  data 
and  5  bytes  of  header),  has  highly  simplified  protocols  (no  win¬ 
dowing  and  no  processor-intensive  work),  incorporates  no  error 
detection  on  the  data  (only  on  the  header),  and  implements  only 
layer  1  and  basic  layer  2  functions  in  the  7-layer  OSI  standard. 
ATM  provides  coimection-oriented  virtual  circuits,  handles  con¬ 
tinuous  and  bursty  data,  elimiiuites  the  need  for  multiple  TDM 
channel  rates,  provides  separate  signal  and  information  channels, 
and  is  independent  of  the  transmission  medium.  ATM  differs  from 
packet  switching  in  the  following  ways:  ATM  has  fixed  length 
cells  (instead  of  variable  length  packets);  ATM  uses  highly  sim¬ 
plified  protocols  (instead  of  processor-intensive  protocols);  ATM 
does  not  do  error  correction  on  the  data  on  a  link-by-litik  basis; 
and  ATM  does  not  do  any  layer  3  operations. 

In  addition,  the  IEEE  802.6  committtee  has  recently  approved 
a  protocol  for  use  in  MAN'S  based  on  the  Distributed  Queue  Dual 
Bus  (DQDB)  [1 1].  This  802.6  MAN  standard  is  compatible  with 
ATM/BISDN  and  provides  a  natural  addition  to  the  emerging 
world  of  Broadband.  The  common  format  shared  among  this 
MAN  standard,  ATM,  and  BISDN  greatly  simplifies  the  inter¬ 
networking  problems  of  the  forthcoming  broadband  era.  Mean¬ 
while,  the  fiber  distributed  data  interface  (FDDI)  has  met  with 
some  success  as  a  lOO-Mb/s  offering  [12]. 

The  carriers  are  beginning  to  offer  their  switched  multimegabit 
data  service  (SMDS)  [13]  which  will  probably  be  the  first  mani¬ 
festation  of  the  802.6  MAN.  SMDS  has  already  been  demon¬ 
strated  at  45  Mb/s  and  will  soon  be  offered  on  a  tariffed  basis. 
SMDS  differs  from  ISDN  in  that  it  is  a  connectionless  data  service 
that  includes  broadcast  and  multicast  features.  ISDN,  on  the  other 
hand,  is  an  integrated  voice  and  data  service  offering  both  circuit- 
switched  and  packet  switched  features. 

A  near-term  problem  we  foresee  for  the  carriers  who  are  to 
offer  these  services  is  the  issue  of  establishing  a  tariff  that  will 
satisfy  the  end  user  in  matching  his  patterns  of  use  in  the  emerg¬ 
ing  applications. 

In  the  next  five  years,  we  can  anticipate  that  X.25  packet 
switching  will  migrate  to  frame  relay,  to  FDDI  and  then  to  the 
802.6  DQDB  via  SMDS,  finally  bringing  us  to  the  ATM/BISDN 
offerings. 

As  these  brave  new  broadband  capabilities  develop,  it  must  be 
understood  that  our  current  networks  are  ill-suited  to  provide  ser¬ 
vices  using  these  increased  bandwidths.  We  must  re-engineer  the 
architecture  of  our  networks  to  accommodate  these  bandwidths, 
a  topic  we  address  in  the  next  section. 

V.  High  Bandwidth  Networking 

Broadband  ISDN  is  the  proposed  foundation  for  wide  area  net¬ 
works  (WAN  s)  that  are  capable  of  supporting  applications  need¬ 
ing  high  speed,  low  latency,  rich  functionality,  and  support  of 
mixed  media  (i.e..  voice,  data,  image,  video,  graphics,  fax.  etc.). 
The  market  demand  for  these  advanced  applications  is  clearly 


growing.  Fuitliennore,  the  core  technologies  to  provide  these  ser¬ 
vices  are  emerging:  high-speed  switches  are  being  designed,  high¬ 
speed  fiber  access  netwoiks  are  being  deployed,  the  SONET  hier¬ 
archy  has  been  defined,  ATM  multiplexing  techniques  are  agreed 
upon,  etc.  Indeed,  technology  is  solving  most  of  the  peifoimance 
problems  we  can  foresee  Gink  speeds,  processor  speeds,  and 
memoiy  sizes  are  increasing  on  their  own). 

As  we  move  into  gigabit  networks,  however,  we  must  take  a 
“clean  sheet"  approach  to  many  of  the  systems  issues  [14],  [15]. 
The  critical  areas  to  be  considered  include  switching  technology, 
processor  interfaces,  protocols,  connection-oriented  communi¬ 
cations,  routing,  layered  architectures,  and  coexistence  with  car¬ 
rier  environments.  We  must  be  prepared  to  allow  different  switch 
technologies  to  work  in  the  future  broadband  networks;  these 
include  the  BISDN  fast  packet  switching  techniques,  photonic 
switches,  and  wave-length  division  multiplexing  (WLDM).  The 
architecture  we  select  must  not  depend  upon  which  of  these  hap¬ 
pens  to  be  implemented. 

As  for  switching,  tomorrow's  netwoiks  must  be  prepared  to 
handle  packet,  circuit,  aiKl  hybrid  switches.  Large  packets  or 
groups  of  packets  will  have  to  be  switched  simultaneously;  at 
gigabit  bandwidths,  one  cannot  afford  the  overhead  of  switching 
small  blocks  independently.  Sophisticated  dynamic  bandwidth 
reservation  algorithms  must  be  developed.  Multicast  algorithms 
and  capabilities  must  be  developed  (fiber  is  point-to-point, 
whereas  satellite  and  ground-based  radio  are  broadcast  and  mul¬ 
ticast). 

Beyond  all  of  these,  the  question  of  the  network  management 
system  is  extremely  important.  Today's  nets  are  reaaive,  not 
proactive.  We  must  introduce  proactive  diagnosis  and  service  re- 
storal  before  users  sense  a  problem.  We  need  proactive  resource 
management.  Since  huge  volumes  of  raw  data  will  be  flowing  into 
the  management  control  center,  we  must  use  thresholds,  filters 
and  alerts,  and  even  expert  systems,  for  early  problem  detection 
and  resolution.  These  management  functions  must  operate  in  a 
distributed  fashion  for  fault  conuinment.  privilege  definition,  and 
localization  of  security  failures.  Multiple  classes  of  service  must 
be  supported.  Adaptive  protocols  and  error  recovery  mechanisms 
must  be  developed.  Indeed,  the  management  of  the  emerging 
internetwork  is  turning  out  to  be  the  ultimate  challenge  in  distrib¬ 
uted  systems. 

As  we  consider  these  problems,  it  is  clear  that  the  carriers  have 
been  facing  large  network  problems  for  most  of  this  century.  They 
understand  management,  billing,  accountability,  security,  avail¬ 
ability.  introduction  of  new  technology  on  a  large  scale,  etc. 
However,  over  the  last  twenty  years,  the  innovations  in  data  net¬ 
working  have  come  from  the  data- processing  industry,  and  not 
from  the  carriers.  (This  in  spite  of  the  fact  that  the  data-processing 
solutions  have  used  the  underlying  carrier  plant  to  establish  their 
data  networks).  As  we  move  into  the  broadband  era,  it  is  essential 
that  these  two  (merged)  industries  cooperate  in  providing  service 
to  the  user  community.  BISDN  holds  much  promise  for  advanced 
networking,  and  the  technological  and  managerial  hurdles  that 
must  be  overcome  are  best  solved  jointly  by  these  two  industries. 

VI.  Conclusions 

The  concept  of  ISDN  was  generated  from  the  carriers.  Its  early 
growth  was  much  slower  than  had  been  promised  due  to  a  number 
of  reasons,  key  among  them  being  the  lack  of  real  user  demand 
for  the  service.  However,  in  the  past  two  years,  the  narrowband 
ISDN  (NISDN)  penetration  has  accelerated  faster  than  the  skep¬ 
tics  had  been  predicting. 
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ISDN  is  the  means  by  which  the  less  advanced  users  can  quickly 
catch  up  to  today's  technology.  However,  the  teal  payoff  will 
come  with  BISDN.  The  dau  network  services  and  capacity  offered 
by  BISDN  are  truly  exciting  and  advanced.  But  we  must  proceed 
with  NISDN  before  we  can  achieve  BISDN. 

The  carriers  have  an  enormous  investment  in  ISDN  and  they 
are  highly  motivated  to  bring  about  its  success.  The  carriers  ate 
the  key  to  the  future  networking  infrastructure  for  the  U  S.  and 
the  test  of  the  world.  The  data-processing  industry  cannot  “go  it 
alone  in  this  endeavor;  they  must  cooperate  and  encourage  the 
carnets.  Both  groups  must  agree  on  common  standards  for  both 
private  and  public  networking  as  this  infrastructure  grows.  ISDN 
is  one  important  step  in  this  direction.  Beyond  that,  however,  it 
must  be  recognized  that  a  revolutionary  approach  must  be  taken 
in  providing  the  gigabit/second  services  about  which  we  are  talk¬ 
ing.  The  fundamental  architecture  of  our  plant  must  be  over¬ 
hauled  significantly;  that  overhaul  is  already  well  underway. 

It  is  perhaps  worthwhile  to  review  some  of  the  economic  fac¬ 
tors  that  have,  and  will,  affect  the  architecture  of  our  communi¬ 
cation  networks.  The  cost  of  moving  data  across  a  network  con¬ 
sists  of  two  important  components;  the  cost  of  the  channels  and 
the  cost  of  the  switches.  In  the  early  days  of  communications,  the 
channel  was  the  expensive  component  (copper  wires  strung  up  on 
telephone  poles)  and  the  switch  was  a  poorly  paid  human  opera¬ 
tor.  As  a  result,  one  could  afford  to  waste  switch  capacity  to  save 
on  the  expensive  communications  component.  Then,  a  revolution 
occurred  in  communications;  microwave  radio  was  introduced  and 
this  dramatically  dropped  the  cost  of  the  communications  com¬ 
ponent.  At  the  same  time,  the  switch  cost  dropped  (automatic 
switches  in  the  form  of  relays  and  vacuum  tubes  appeared),  but 
not  as  dramatically  as  the  channel.  Consequently,  a  reversal 
occurred  where  the  switch  was  now  more  expensive  than  the  com¬ 
munication  channel.  Now  it  was  sensible  to  waste  communica¬ 
tions  capacity  in  order  to  save  on  the  switch.  Thus  circuit  switch¬ 
ing  was  introduced.  In  the  1970’s,  another  reversal  occurred  when 
integrated  electronics  (VLSI)  appeared,  which  dramatically 
dropped  the  switch  cost  relative  to  the  communications  cost.  Once 
again,  we  could  afford  to  waste  switching  capacity  in  order  to 
save  yet  more  on  the  communications  costs.  Thus  packet  switch¬ 
ing  was  introduced. 

That  was  the  past.  Let  us  now  peek  into  the  future.  Is  there 
anything  out  there  in  the  near  term  that  will  dramatically  drop  the 
cost  of  the  switch?  Gallium  arsenide  components  will  help,  but 
they  do  not  represent  a  revolutionary  change.  On  the  other  hand, 
warm  superconductivity,  if  it  comes,  would  indeed  be  a  dramatic 
improvement  in  switch  technology.  It  would  allow  the  wires  to 
be  thinner  (and  still  not  generate  much  heat)  thereby  allowing 
smaller  dimensions  (i.e. ,  reduced  latency  due  to  the  speed  of  light) 
and  tighter  packing.  However,  warm  superconductivity  is  not  a 
near-term  likelihood.  Further,  photonic  switching  would  be  a  rev¬ 
olutionary  improvement  in  switch  technology.  Here,  too,  we  are 
talking  about  a  laboratory  experiment  and  not  a  near-term  devel¬ 
opment.  So  the  answer  is  "no”;  we  cannot  foresee  a  dramatic 
improvement  in  switch  technology  near  term.  But  how  about  a 
revolution  in  communications?  Is  there  a  technology  out  there 
that  will  dramatically  reduce  the  cost  of  communications?  The 
answer  is  a  resounding  "YES!”  Indeed  it  is  already  taking  place, 
and  it  is  called  fiber  optics.  As  stated  above,  we  have  well  over 
a  million  miles  of  fiber  optics  in  place  in  the  U.S.  alone.  We  are 
in  the  midst  of  the  next  reversal,  which  leads  us  to  a  situation 
where  communications  ate  plentiful  and  the  bottleneck  has  once 
again  become  the  switch.  Our  networking  architectures  are 
undergoing  a  massive  revamping  as  we  move  into  this  environ- 


mem.  Our  I980's  architectures  ate  inadequate  for  the  economics  ) 

and  applications  of  the  1990's. 

In  response  to  this  current  reversal,  we  sec  BISDN  services 
coming  along,  we  see  fast  packet  switching  architectures,  we  see 
ATM,  we  see  the  802.6  MAN,  we  see  LAN  developments,  we 
see  FDDI,  etc.  And,  once  we  get  all  that  wonderful  technology 
in  place,  is  it  possible  that  either  warm  superconductivity  and/or 
photonic  switching  will  come  along  so  as  to  cause  yet  a  further 
reversal  and  thus  another  reshuffling  of  the  cards?  It  seems  there 
will  be  a  need  for  continual  improvement  of  architectures  and 
systems  as  new  technological  developments  spawn  new  possibil¬ 
ities  and  new  applications. 

As  we  begin  to  move  through  the  1990’s  we  forsee  that  broad¬ 
band  ISDN  will  play  an  important  role  in  bringing  about  some  of 
the  exciting  networking  developments.  A  great  deal  of  research 
has  gone  into  broadband  networking  in  the  last  few  years.  The 
next  few  years  will  see  development  of  products  and  growth  in 
demand.  There  is  no  question  but  that  this  technology  will  pro¬ 
vide  the  basis  for  a  ubiquitous  communications  infrastmctute  of 
enormous  capacity. 

Let  us  conclude  this  paper  by  listing  some  of  the  components 
that  we  are  likely  to  see  in  this  time  frame: 

•  Worldwide  Data  Networks 

•  Advanced  Network  Machines 

•  Optical  Fiber  Networks 

•  Gigabit /second  Networks 

•  Megapacket/s  Superswitches 

•  Optical  Switches 

•  Pervasive  Local  Area  Networks 

•  LAN-MAN-WAN  Hierarchy 

•  Processing  Satellites 

•  Intelligent  Network  Directories 

•  Continuous  Speech  Recognition 

•  Image  Communication  Mode 

•  Digital  Signal  Compression 

•  Massively  Parallel  Systems 

•  Massively  Connected  Systems 

•  Neural  Networks 

•  Pervasive  Expert  Systems. 

It  is  clear  from  this  list  that  the  convergence  of  data  processing 
and  data  communications  is  virtually  complete.  Distributed  infor¬ 
mation  networks  are  poised  to  provide  the  many  services  required 
for  the  emerging  information  society.  ISDN  will  serve  to  hasten 
access  to  these  information  networks,  eventually  providing  a 
major  thrust  when  BISDN  products  and  services  begin  to  toll  out. 
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